Every now and then I dig into PHP internals with gdb, typically to debug issues with the Gearman Pecl extension as I’m currently primary maintainer for the project. It’s a fairly low time demand, despite the handful of issues lingering at the moment, so I’ll poke at it maybe 3 or 4 times a year since completing the migration to PHP 7 and adding some tests in 2015.
That migration was fun as it was my first foray into any underlying code that powered PHP. I’ve worked with C and C++ in years past, at least enough to get by, but never so much that I’d call myself a pro or fluent enough that I could hop in without immediately needing to reference documentation. This was a hard leveling up moment for me, a challenge to move out of my comfort zone and admit “wow, there’s a lot here I just don’t know”.
As there’s often a gap of several months every time I look at PHP under the hood, that muscle memory isn’t always as strong as I’d like, having to jump back to notes of my own when I open up an issue for investigation. I also think it’s important that, as engineers of any level, we reaffirm that we don’t know everything, or have it able to recalled at a moment’s notice. There’s absolutely nothing wrong with having to write notes to remember things that are “obvious” to other folks.
With that in mind, here are a few things I’ve learned about debugging PHP with gdb.
A trivial function in the Gearman pecl extension
Let’s look at a simple PHP function in the Gearman extension:
PHP_FUNCTION(gearman_verbose_name) { zend_long verbose; if (zend_parse_parameters(ZEND_NUM_ARGS(), "l", &verbose) == FAILURE) { php_error_docref(NULL, E_WARNING, "Unable to parse parameters."); RETURN_NULL(); } if (verbose < 0) { php_error_docref(NULL, E_WARNING, "Input must be an integer greater than 0."); RETURN_NULL(); } RETURN_STRING((char *)gearman_verbose_name(verbose)); }
If you’ve never looked at the underlying PHP code or even C code before, it can be slightly daunting (there are quite a few macros to get your head wrapped around). A quick break down of the above function:
- Define a function called gearman_verbose_name.
- Create a variable called verbose, where a zend_long can be thought of in more simplistic terms as an integer (a long integer more specifically, with some PHP internal object stuff baked in to it).
- The first conditional is looking for an integer to be passed in when the function is called in PHP, specifically a long. The zend_parse_parameters function is internally defined in the PHP core. It’s requiring one and only one long passed in. If not, throw an error with “Unable to parse parameters.” as the message and return null.
- If a long was passed in, store it in the verbose variable defined earlier. We’re passing a reference of this variable in.1
- The next conditional looks to do some sanity checking on that variable. We should never be getting integers below 0, so throw a warning with a different error message if so.
- If everything is right, call the function gearman_verbose_name (be careful of the overloaded name) which is internal to the gearman library, passing in this long. Cast the value returned as a c-string and return it to be used in the PHP code.
The values that can be passed in are defined elsewhere in the libgearman code as an enum, but for clarity they are:
GEARMAN_VERBOSE_NEVER GEARMAN_VERBOSE_FATAL GEARMAN_VERBOSE_ERROR GEARMAN_VERBOSE_INFO GEARMAN_VERBOSE_DEBUG GEARMAN_VERBOSE_CRAZY GEARMAN_VERBOSE_MAX
The corresponding function in the libgearman code will output a value for each as a string. A one liner in PHP executing this might look something like:
<?php print gearman_verbose_name(GEARMAN_VERBOSE_DEBUG) . PHP_EOL;
With the output being:
DEBUG
The value of this constant is 4, which you can see by simple printing it in PHP
print GEARMAN_VERBOSE_DEBUG;
This is all fairly trivial to accomplish directly within the C code extension without having to reference an underlying library as well, but makes for a good example of tying a library in C code to PHP with the extension as a go between.
Running GDB with PHP
Say there were some underlying problem passing the value of this constant from PHP through the wrapper and to the libgearman code, though. We want to do some inspection of what’s happening with this verbose variable we define early in our code. We can use gdb to walk through code and poke around. To do so, we tell gdb to use the php program in this session. We can do so with the following:
gdb php
With that, we should see some basic info about gdb – the version currently installed, license info, how to use the “help” command, etc. The prompt should also change to (gdb).
GNU gdb (GDB) 7.9 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from php...done.
So gdb is running and it’s using PHP. This last line is interesting too – “Reading symbols from php…done”. We’ll keep that in our back pocket for a bit. What can do we now? We have to tell gdb which php code we want to run, such as the following:
(gdb) run example.php
If example.php took command line arguments, we could pass those in here as well:
(gdb) run example.php arg1 arg2 ...
Note: if we want to exit, we tell it to quit (or just q for short).
That’s great, but everything’s happening so fast. It prints out some info about what’s running, does its work and gives an exit message. We’re not doing very much in our example code and there aren’t any errors, so it exits normally.
Working with breakpoints
Let’s give gdb something to work with by inserting a breakpoint in the gearman pecl extension code we have above (including line numbers for clarity):
921 PHP_FUNCTION(gearman_verbose_name) {
922 zend_long verbose;
923
924 if (zend_parse_parameters(ZEND_NUM_ARGS(), "l",
925 &verbose) == FAILURE) {
926 php_error_docref(NULL, E_WARNING, "Unable to parse parameters.");
927 RETURN_NULL();
928 }
929
930 if (verbose < 0) {
931 php_error_docref(NULL, E_WARNING, "Input must be an integer greater than 0.");
932 RETURN_NULL();
933 }
934
935 __asm__("int3");
936
937 RETURN_STRING((char *)gearman_verbose_name(verbose));
938 }
On line 935, you can see we’ve added a new instruction. This will indicate a breakpoint, specifically a SIGTRAP signal that indicates to a debugger it should be caught and the execution of the program paused for further inspection. If you run this outside of gdb, it’ll print a message about the breakpoint:
> php example.php Trace/breakpoint trap
Since we’re not in the context of a debugger, the program ends. Let’s try running this again in gdb:
(gdb) run example.php Starting program: /usr/local/bin/php example.php [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Program received signal SIGTRAP, Trace/breakpoint trap. zif_gearman_verbose_name (execute_data=0x7fffef2130a0, return_value=0x7fffef213080) at /home/vagrant/gearman/pecl-gearman/php_gearman.c:937 937 RETURN_STRING((char *)gearman_verbose_name(verbose));
Now we have lots of information here! Of particular note:
- zif_gearman_verbose_name indicates we’re currently in the gearman_verbose_name function. “zif_” is a prefix for functions we define in PHP which I believe (please correct me if I’m wrong, internet!) is short for “zend internal function”.
- The code we’re in can be found in the file php_gearman.c, where gearman_verbose_name is defined.
- The program has stopped on line 937. The trap is defined on 935 and there’s a blank line on 936, but the program halts on the line about to be executed, which is the return statement on 937.
This is fun, but is it telling us anything we didn’t know before? We put that code in on line 935 and it stopped where we knew it would. The fun part is that the program is still running! We can see the state of our program and inject stuff along the way.
We began talking about gdb by assuming a case where our function wasn’t working and that the timeout from PHP wasn’t being passed correctly to the libgearman C code. At this point in the code, we can take a look at what the value of that verbose variable is by printing it out with the print command:
(gdb) print verbose $1 = 4
This makes sense. We passed in GEARMAN_VERBOSE_DEBUG, which has a value of 4, in the PHP and the verbose var in the extension was assigned that value. We could do the same with any variables that are in scope at the time the program is halted.
An alias for the print command is also just the keyword p. If we’re looking to see the variable’s address in memory, we can reference it or if it were a pointer, we could dereference it:
(gdb) p &verbose $2 = (zend_long *) 0x7fffffffadb0 (gdb) p *$2 $3 = 4
You can see in the second command, we don’t even use variables defined in the code, but each of the values asserted are stored in local variables.
Note: if we were to try to quit now, we’d get a warning with a prompt whether we were sure we really wanted to quit, as the program isn’t finished yet
(gdb) q A debugging session is active. Inferior 1 [process 7577] will be killed. Quit anyway? (y or n)
If you program has outside dependencies (an open transaction to a database, for example) there may be unintended consequences for exiting if can’t handle suddenly being closed (like the data being rolled back). In a simple program like this, it shouldn’t be harmful to your system.
Moving forward after a breakpoint
If you want to resume your program from this point forward, you can use the continue (or c for short) keyword to move past the breakpoint. In this case, there are no more breakpoints, sot he program finishes successfully.
(gdb) p verbose $1 = 4 (gdb) continue Continuing. DEBUG [Inferior 1 (process 7577) exited normally]
Keeping an eye on when an object changes
What if we wanted to see how a value changed at various points within its scope? Let’s modify our PHP extension by moving the trap point higher in the function, after the verbose var is defined, but before it is initialized:
921 PHP_FUNCTION(gearman_verbose_name) {
922 zend_long verbose;
923
924 __asm__("int3");
925
926 if (zend_parse_parameters(ZEND_NUM_ARGS(), "l",
927 &verbose) == FAILURE) {
928 php_error_docref(NULL, E_WARNING, "Unable to parse parameters.");
929 RETURN_NULL();
930 }
931
932 if (verbose < 0) {
933 php_error_docref(NULL, E_WARNING, "Input must be an integer greater than 0.");
934 RETURN_NULL();
935 }
936
937 RETURN_STRING((char *)gearman_verbose_name(verbose));
938 }
Running this in gdb, the output would look like:
Program received signal SIGTRAP, Trace/breakpoint trap. zif_gearman_verbose_name (execute_data=0x7fffef2130a0, return_value=0x7fffef213080) at /home/vagrant/gearman/pecl-gearman/php_gearman.c:926 926 if (zend_parse_parameters(ZEND_NUM_ARGS(), "l",
and printing out verbose would look like:
(gdb) p verbose $1 = 24848336
Where’d this value come from? Well, since we haven’t initialized it, the value is indeterminate, so it’s not to be trusted. We can use the watch keyword on this variable then continue, which will halt if the variable changes and print its new value:
(gdb) watch verbose
Hardware watchpoint 1: verbose
(gdb) c
Continuing.
Hardware watchpoint 1: verbose
Old value = 24848336
New value = 4
0x0000000000ae8618 in zend_parse_arg_long (arg=0x7fffef2130f0, dest=0x7fffffffadb0, is_null=0x0, check_null=0, cap=0)
at /home/vagrant/php-src/Zend/zend_API.h:1093
1093 {
If you try to attach a watch to an object that is not yet defined or no longer in scope, you’ll receive an error.
Walking through individual source lines
Sometimes we don’t want to have to set tons of halt statements everywhere. Maybe we want to continue, one source line at a time. We can use the step keyword (or s for short) to walk through our code, albeit slowly:
(gdb) s
1102 } else if (cap) {
(gdb) s
1103 return zend_parse_arg_long_cap_slow(arg, dest);
(gdb) s
zend_parse_arg_impl (arg_num=1, arg=0x7fffef2130f0, va=0x7fffffffacc0, spec=0x7fffffffac30, error=0x7fffffffabc0, severity=0x7fffffffabbc)
at /home/vagrant/php-src/Zend/zend_API.c:534
534 /* scan through modifiers */
(gdb) s
742 return "";
(gdb) s
744 break;
(gdb) s 3
778 static int zend_parse_arg(int arg_num, zval *arg, va_list *va, const char **spec, int flags) /* {{{ */
You can see it’s walking through code not just within the gearman extension, but the underlying PHP core library as well. In the last line, I run s 3 which is saying step three times, not just once.
Reviewing the current stack trace
At this point, we might be a little lost as to where we are within our program, stepping so slowly and into the PHP core library. We can use the backtrace (or bt) command to give ourselves a backtrace of the function stack.
(gdb) bt
#0 zend_parse_arg (arg_num=1, arg=0x7fffef2130f0, va=0x7fffffffacc0, spec=0x7fffffffac30, flags=0) at /home/vagrant/php-src/Zend/zend_API.c:778
#1 0x0000000000aebcf4 in zend_parse_va_args (num_args=0, type_spec=0x7fffeeb00b3a "", va=0x7fffffffacc0, flags=0)
at /home/vagrant/php-src/Zend/zend_API.c:926
#2 0x0000000000aebe9f in zend_parse_parameters (num_args=1, type_spec=0x7fffeeb00b39 "l") at /home/vagrant/php-src/Zend/zend_API.c:960
#3 0x00007fffeeaf6202 in zif_gearman_verbose_name (execute_data=0x7fffef2130a0, return_value=0x7fffef213080)
at /home/vagrant/gearman/pecl-gearman/php_gearman.c:926
#4 0x0000000000b46c77 in ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER (execute_data=0x7fffef213030) at /home/vagrant/php-src/Zend/zend_vm_execute.h:664
#5 0x0000000000b46345 in execute_ex (ex=0x7fffef213030) at /home/vagrant/php-src/Zend/zend_vm_execute.h:432
#6 0x0000000000b46496 in zend_execute (op_array=0x7fffef274300, return_value=0x0) at /home/vagrant/php-src/Zend/zend_vm_execute.h:474
#7 0x0000000000ae76e3 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /home/vagrant/php-src/Zend/zend.c:1541
#8 0x0000000000a56f0e in php_execute_script (primary_file=0x7fffffffe360) at /home/vagrant/php-src/main/main.c:2535
#9 0x0000000000bd4443 in do_cli (argc=2, argv=0x172ad80) at /home/vagrant/php-src/sapi/cli/php_cli.c:997
#10 0x0000000000bd5412 in main (argc=2, argv=0x172ad80) at /home/vagrant/php-src/sapi/cli/php_cli.c:1390
We can see at the top it starts with the inner most function we’re in and at the bottom, the main function in the example.php script.
Remember earlier how we put a pin in loading symbols? When gdb starts, it loads the php debug package along with the main php core library. If you see lots of ???? in your backtrace, it means it hasn’t loaded this symbol library. The php core has a debug library already built to download as do most extensions. If you’re rolling PHP yourself, make sure to include the –enable-debug option.
Printing the source
You can also use the list command (alias l, a lower case L) to list 5 lines before and 5 after the current line, or give it a number to show 5 lines before and after that given line
(gdb) list 10 5 | Copyright (c) 1998-2017 Zend Technologies Ltd. (http://www.zend.com) | 6 +----------------------------------------------------------------------+ 7 | This source file is subject to version 2.00 of the Zend license, | 8 | that is bundled with this package in the file LICENSE, and is | 9 | available through the world-wide-web at the following url: | 10 | http://www.zend.com/license/2_00.txt. | 11 | If you did not receive a copy of the Zend license and are unable to | 12 | obtain it through the world-wide-web, please send a note to | 13 | license@zend.com so we can mail you a copy immediately. | 14 +----------------------------------------------------------------------+
Using gdb on core dumps and running processes
This is all convenient when you have a script ready to run, but what happens if you want to catch errors, like segfaults, after the fact? You can do so by generating core dumps whenever segfaults hit. You’ll need to:
- remove limits on core dump size. For bash or sh, you can do this with:
ulimit -c unlimited
or in tcsh:
unlimit coredumpsize
There’s ways for setting this in other environments but I won’t exhaustively list them here. This also means core dump sizes can grow very large! Be careful you don’t run out of disk space.
- Set the core directory (running as root):
echo "<cores dir>/core-%e.%p" > /proc/sys/kernel/core_pattern
This is how your OS knows where to write the core files to and the format for the file names. Make sure you have perms to write to the core dir, or if you’re running PHP as a web app that process has perms.
- Restart your webserver as well if you’re running it in, say apache, as the server will need to pick up these changes. A graceful reload won’t be sufficient here, though.
You can then walk through the core dump with:
gdb php <path/to/core/file>
If the core file was generated via apache, you’ll want to run this as:
gdb <path/to/httpd> <path/to/core/file>
When the gdb prompt appears, run the backtrace command to find where the segfault hit.
One final tip – if the process is already running, you can attach gdb to it with the following:
gdb <program name> -p <pid>
but make sure you have the same permissions as the running process and be very careful about doing this in a production environment, as this can be harmful to live requests (slowing or stopping, modifying values, etc.).
Conclusion
There are so many more useful commands to run in gdb, tips for setting up cores, or digging into failure modes in PHP. We can edit values in real time, continue until we hit a specific line or function, change the flow control, etc. with gdb. It is immensely powerful, though the learning curve can be steep. These are just a few of the commands I run often enough. I may add to this post or create follow up posts when I pick up other tasks, but if you have additional tips and tricks, please pass them along as well. I’m hoping this serves not only as a good refresher for me and reference for anyone else looking to pick this up, but a way for me to learn something new too.
Update 1:
Totally forgot to include this. Big thanks to Johannes Schlüter who mentioned .gdbinit. It’s a powerful set of tools to include with your gdb debugging that I’ve barely scratched the surface on. If I write a follow up, that’s a great place to start, but don’t wait for me before looking into it yourself.
1 comment for “A 101 on debugging php internals with gdb”