In MariaDB 5.2 there are show table_statistics show index_statistics, and show user_statistics functions that return global stats collected by the server when the userstats variable is enabled. There are also associated information schema tables that feed these commands. So far I’ve changed the syntax of the show command to be

SHOW [ QUERY | SESSION | GLOBAL ] TABLE_STATISTICS

Where the default show table_statistics returns the global stats. This is different from the tivo patch against 5.0.72 which changed the default to be the last query. The new syntax keeps things backwards compatible with the original show command. The query keyword will return the previous query_statistics_history_size queries of statistics. This is a bit different from the 5.0 version which only tracked one query.

For example if I set query_statistics_history_size=5 then mariadb will track the row stats from the previous 5 queries:

mysql> set query_statistics_history_size=5;
Query OK, 0 rows affected (0.00 sec)

mysql> show tables;
+—————-+
| Tables_in_test |
+—————-+
| t |
| t2 |
+—————-+
2 rows in set (0.00 sec)

mysql> select * from t;
+—+
| t |
+—+
| 1 |
| 2 |
| 3 |
| 4 |
| 5 |
+—+
5 rows in set (0.00 sec)

mysql> select * from t2;
+—-+
| t2 |
+—-+
| 1 |
| 2 |
| 3 |
+—-+
3 rows in set (0.00 sec)

mysql> show query table_statistics;
+———-+————–+————+———–+————–+————————-+
| Query_id | Table_schema | Table_name | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+———-+————–+————+———–+————–+————————-+
| 81 | test | t | 5 | 0 | 0 |
| 82 | test | t2 | 3 | 0 | 0 |
+———-+————–+————+———–+————–+————————-+
2 rows in set (0.00 sec)

mysql> select count(*) from t,t2;
+———-+
| count(*) |
+———-+
| 15 |
+———-+
1 row in set (0.00 sec)

mysql> show query table_statistics;
+———-+————–+————+———–+————–+————————-+
| Query_id | Table_schema | Table_name | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+———-+————–+————+———–+————–+————————-+
| 81 | test | t | 5 | 0 | 0 |
| 82 | test | t2 | 3 | 0 | 0 |
| 84 | test | t | 5 | 0 | 0 |
| 84 | test | t2 | 3 | 0 | 0 |
+———-+————–+————+———–+————–+————————-+
4 rows in set (0.00 sec)

Note that show query table_statistics shows the stats only for the current connection. There isn’t a way to get lists from other threads. My next task is to add a separate table with the query_id and query text similar to show profiles that can be used to correlate a query back to the different statistics kept for it. After that I plan on adding index statistics.

The older version of this in the tivo patch has been extremely useful for optimizing queries. I hope I can make this even better and get it into mariadb for other people to use. If you have any suggestions for how I should change this please post in the comments.

mysqld_safe modifications

–fallback-ledir
The –ledir option tells mysqld_safe which directory contains the mysqld file that it should use. TiVo has added the –fallback-ledir option which will switch the ledir should mysqld crash. This makes it possible to run a new mysqld binary then fallback to the stock one should the new one crash. This feature has never been called to duty in production but it makes me sleep better.

–crash-script
This is a new option to mysqld_safe which will execute a script when it detects mysqld has crashed. This can be useful for gathering diagnostic info, sending monitoring alerts or even triggering a failover. Remember if you choose to use this option to trigger a failover make sure your failover mechanism has flap prevention so the master won’t ping/pong back and forth between servers.

mysqldump modifications

–slave-data
I think this option has been implemented by several people. We have also added it because it makes life easier for making backups. What this does is mark the replication position of the slave thread in the mysqldump output. It’s also useful for creating a dump file from a slave and using it to restore a peer slave.

–log-pos-comment
This makes dump files a little bit friendlier for restoring backups on one of an HA pair of machines. It writes out a pair of comments with replication positions from both the slave thread and the master position in a human readable format. This is handy for debugging restore scripts as well as those rare situations when a backup needs to be restored by hand.

–no-bin-log
This tells mysqldump to write a SET SQL_LOG_BIN=0; to the top of the dump file in the same way other variables are set. This makes dump files import only on the machine and not be written to the replication stream. It makes dump files a bit safer for importing on an HA pair. TiVo has multiple tools that work with sqldump files so this was a good mechanism to make the dump files replication safe.

The rest of the modifications are in mysqld itself. TiVo makes extensive use of the row stats options originally created at Google. The command syntax has been modified from Google’s original version to make it easier for developers to use the commands to debug queries as well as allowing the application servers themselves to access the data per session or per query.

TiVo has complicated MySQL queries, really complicated. Sometimes it’s difficult to figure out which part of the query is doing the most work or what changed the performance between two different versions of a query. One way to help solve that problem is to keep track of the rows being accessed in a query both for that query and for the session.

This description applies for both SHOW TABLE_STATISTICS and SHOW INDEX_STATISTICS commands.

Row stats in the TiVo patch are tracked at 3 different levels, per query, per session, and globally. The original Google patch only tracked row stats globally. Tracking them per session and per query doesn’t add much overhead and provides valuable information about query performance. It’s true that not every row access is the same but this has provided an easier measure for work done by a query than wall clock time or even rows examined from the slow query log.

After a query has executed the row statistics for that query can be accessed with:

mysql> select * from t;
+------+
| t    |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql> show table_statistics;
+--------+-----------+--------------+-------------------------+
| Table  | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+--------+-----------+--------------+-------------------------+
| test.t |         3 |            0 |                       0 |
+--------+-----------+--------------+-------------------------+
1 row in set (0.00 sec)

If the query is executed again show table_statistics will still only return results for the previous query but show session table_statistics will return cumulative counts for the duration of the session. The rows changed in the session query is from the insert statement which I used to create the test table.

mysql> select * from t;
+------+
| t    |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql> show table_statistics;
+--------+-----------+--------------+-------------------------+
| Table  | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+--------+-----------+--------------+-------------------------+
| test.t |         3 |            0 |                       0 |
+--------+-----------+--------------+-------------------------+
1 row in set (0.00 sec)

mysql> show session table_statistics;
+--------+-----------+--------------+-------------------------+
| Table  | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+--------+-----------+--------------+-------------------------+
| test.t |         6 |            3 |                       3 |
+--------+-----------+--------------+-------------------------+
1 row in set (0.00 sec)

The final way to access the row statistics is the global count for the server. All of these commands also support using like syntax to narrow down the list of tables. In this case I’m showing the global stats which include the mysql tables.

mysql> show global table_statistics like 'mysql%';
+----------------------+-----------+--------------+-------------------------+
| Table                | Rows_read | Rows_changed | Rows_changed_x_#indexes |
+----------------------+-----------+--------------+-------------------------+
| mysql.proc           |       293 |            4 |                       4 |
| mysql.user           |  10237928 |            2 |                       2 |
| mysql.help_category  |        36 |            0 |                       0 |
| mysql.help_topic     |       463 |            0 |                       0 |
| mysql.help_relation  |       733 |            0 |                       0 |
| mysql.help_keyword   |       381 |            0 |                       0 |
| mysql.db             |  10964886 |            0 |                       0 |
+----------------------+-----------+--------------+-------------------------+
8 rows in set (0.00 sec)

Since these statistics are readily available within MySQL the TiVo patch modifies the slow query log header to include table and index statistics.

# Time: 100725 14:39:33
# User@Host: tivo[tivo] @ localhost []
# Query_time: 3  Lock_time: 0  Rows_sent: 3  Rows_examined: 3
# Row_Stats: test.t:rows_read=3,rows_changed=0,rows_changed_x_indexes=0;
# Index_Stats:
select *, sleep(1) from t;

The TiVo patch also includes changes to InnoDB from the Google patch and from Percona.

This means that Drizzle is no longer a side project hacked on by a few engineers outside of their work time, it’s an official project. With Sun employing Drizzle engineers and Sun owning the copyright to the MySQL documentation it’s now likely that Sun will create Drizzle docs from the existing MySQL docs to support their new product. The question now is how will the docs be licensed? Will Drizzle enjoy the fork protection that MySQL has for so many years by keeping it’s documentation hostage? Probably. If Sun creates docs based on the MySQL docs they will own the copyright and effectively own Drizzle.

The trick is to run through this sequence twice. Turn on the headphones by holding the button on the right ear. After you hear the ping Mac OS will prompt you to use the headphones. Choose the option to use the headphones and the bluetooth icon will switch to connected. Now turn on the headphones again and when you hear them ping the bluetooth icon should switch back to disconnected. Under the bluetooth menu choose “Use Headphones” under the FreePulse device. The icon will stay disconnected but when you play audio it will switch to connect and start functioning.

I couldn’t find a place on apple’s site to report the bug with repeatable test case. If any of you know how to report bugs in Mac OS please tell me and I’ll file it so hopefully this behavior is fixed in future releases.

Hybrids are fine for city driving but they offer no improvement over your typical four-banger once you get out on to the highway or back in the woods. Let’s face it, most hybrids have no business off pavement. In a few years I’m sure decent hybrid trucks will become available. I’m not holding my breath.

The other alternative is electric cars. They’re great for the daily commute. You drive to work and back then plug them in when you get home a night. Electric cars are great because we have so many different clean options for generating power. The typical range is 200 miles so they have plenty of juice to go to work and grab groceries on the way home. If you need to drive outside that range you’re effectively screwed. After the juice runs out the car has to be plugged in for hours before you can make go another 200 miles.

In order to get people to switch over to all electric cars we need to have the range of a gas fueled engine plus the ease of refueling. Since we can’t recharge batteries as fast as we can fill a tank with gas the only other option is to change the batteries out. I think we need to standardize on battery size and have them removed from the bottom of the car. Then instead of gas stations we can have automated robots that will drop all the batteries out of the bottom of a car and place charged batteries in. Think of it like driving through an automated car wash like device but instead of cleaning your car it drops the batteries out of the bottom and puts new ones in. This eliminates two problems with electric cars. First it means we can recharge electric cars as fast as or faster than gas cars and as long as there are battery change out stations we can continue to go on road trips. It also means that there is no longer an issue with the huge cost of replacing batteries after a few years. Part of the recharge cost will also be maintenance on the batteries. It’s like the propane exchange at your local grocery store.

After working on the problem for a bit I found that after jmap ran ps showed the java process as stopped. This is strange since java was still able to process requests. In linux threads are treated as processes, they get a pid just like any other process. To be POSIX compliant linux has the notion of thread groups and a thread group leader so signals can be delivered to an entire thread group.

[Update: 2009-05-19 The version of jmap that ships with jdk-1.6 detaches correctly]

jmap gets memory statistics by using ptrace on the pid of the thread group leader. When ptracing a thread group leader only that thread is stopped and analyzed. Other threads are free to continue processing. Jmap is a bit buggy in that it attaches to a thread but never detaches. Linux has a safe guard that if the parent of a traced process quits then linux changes the traced processes’ state from traced to stopped because traced processes can’t be killed.

When jmap quits the ps shows the process in state T which means stopped. What it doesn’t say is that only the thread group leader is stopped. To get around the limitations of ps and process states I went directly to proc to get the process state. The example below was done using mysqld instead of java. It shows the process state of all the threads, including the leader during a simulated jmap run.

In this example 20924 is the pid of mysqld. I substituted mysql for java in this example because I had it handy on my dev server. It reacts the same way java does. The bad_trace app simulates jmap by doing a ptrace attach and exiting before a detach. It sleeps for a big in the middle so I can get the process state of a normal traced process. Here is the source for bad_trace if you want to follow along at home.

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/ptrace.h>

return 0;
}

The bad_trace app does a ptrace attach and exits without detaching.

This is the normal state on an idle server all threads are sleeping.

ebergen@kamet:(/proc/20924/task) grep State */status
20924/status:State: S (sleeping)
20926/status:State: S (sleeping)
20927/status:State: S (sleeping)
20928/status:State: S (sleeping)
20929/status:State: S (sleeping)
20931/status:State: S (sleeping)
20932/status:State: S (sleeping)
20933/status:State: S (sleeping)
20934/status:State: S (sleeping)

I execute bad_trace which puts the thread group leader into traced mode waiting for bad_trace to examine it.

ebergen@kamet:(/proc/20924/task) ~/bad_trace 20924
Attaching to pid (20924)
Sleeping for ten seconds
[1]+ Stopped ~/bad_trace 20924

I suspend bad_trace to grab the stats. You can see that the thread group leader has stopped in tracing mode waiting for bad_trace to examine it and tell it to continue.

ebergen@kamet:(/proc/20924/task) grep State */status
20924/status:State: T (tracing stop)
20926/status:State: S (sleeping)
20927/status:State: S (sleeping)
20928/status:State: S (sleeping)
20929/status:State: S (sleeping)
20931/status:State: S (sleeping)
20932/status:State: S (sleeping)
20933/status:State: S (sleeping)
20934/status:State: S (sleeping)
ebergen@kamet:(/proc/20924/task) fg
~/bad_trace 20924

bad_trace resumes and exits without detaching from the traced process. To enable an admin to kill the traced process linux does some cleanup work by changing it’s state from traced to stopped. Now we can send signals to the thread group and they will be able to respond.

ebergen@kamet:(/proc/20924/task) grep State */status
20924/status:State: T (stopped)
20926/status:State: S (sleeping)
20927/status:State: S (sleeping)
20928/status:State: S (sleeping)
20929/status:State: S (sleeping)
20931/status:State: S (sleeping)
20932/status:State: S (sleeping)
20933/status:State: S (sleeping)
20934/status:State: S (sleeping)

Naturally we don’t want the thread group leader stopped if everything is OK so I send it a continue signal to resume operation. This is where things get weird.

ebergen@kamet:(/proc/20924/task) kill -CONT 20924

Instead of the thread group leader resuming operation linux decides that it’s a good idea to stop all threads instead.

ebergen@kamet:(/proc/20924/task) grep State */status
20924/status:State: T (stopped)
20926/status:State: T (stopped)
20927/status:State: T (stopped)
20928/status:State: T (stopped)
20929/status:State: T (stopped)
20931/status:State: T (stopped)
20932/status:State: T (stopped)
20933/status:State: T (stopped)
20934/status:State: T (stopped)

Sending a second continue signal does the right thing and the process resumes.

ebergen@kamet:(/proc/20924/task) kill -CONT 20924
ebergen@kamet:(/proc/20924/task) grep State */status
20924/status:State: S (sleeping)
20926/status:State: S (sleeping)
20927/status:State: S (sleeping)
20928/status:State: S (sleeping)
20929/status:State: S (sleeping)
20931/status:State: S (sleeping)
20932/status:State: S (sleeping)
20933/status:State: S (sleeping)
20934/status:State: S (sleeping)

I did some digging in the kernel and didn’t see any specific reason for this behavior. I suspect it’s a kernel bug that has never been uncovered because tracing a single thread of a threaded process isn’t a very common operation.

Currently a flush logs command tries to flush all of the following logs in order:

• General Log
• Slow Query Log
• Binary Log
• Relay Log
• Store Engine Logs (If available)
• Error Log

The reason I wanted to fix this is because my client was issuing a flush logs to rotate the error log on a server with no replication. The crash was caused by replication. With individual flush logs it’s less likely for this to happen again in the future. People can simply issue a query for the log they want to flush. The new commands flush logs named in the command. They are:

• flush general log;
• flush slow log;
• flush binary log;
• flush relay log;
• flush engine logs;
• flush error log;

The words log and logs are interchangeable. The query “flush general log” is just as valid as “flush general logs” even though there is only one log. I submitted the patch as a fix for MySQL Bug #14104.

The patch, flush_logs.patch was diffed against 6.0.4 but also applies on 5.1.24.

Rotation for different log files isn’t uniform. Rotating the slow log simply closes and opens it. I’m planning to write a second patch that rotates log files using the same numbered scheme as binary logs. This fixes the rotation for slow and general log as well as eliminating the annoying issue of error logs being destroyed after they are rotated to foo.log-old.

This patch hasn’t been accepted or committed yet so if you have any suggestions on how to make it better please let me know.

Why didn’t you just put the sticker with the serial number on the manual?

Here is my rant about changing the default value which I won’t repeat here.

Show @&!# status again!