BlueGecko Denmark

Remote DBA for Drizzle

snovotny — Wed, 13 Apr 2011 14:10:25 +0000

This morning at the MySQL User Conference and Expo, we’re excited to be included in Brian Aker’s keynote “The State of Drizzle” as one of the first organization to offer support for Drizzle now that it has reached GA with the release of Drizzle 7. Brian Aker is the CTO of Data Differential and is the lead developer for the Drizzle Project.

Blue Gecko will offer operational support via our remote DBA model for Drizzle environments in the same manner and with the same stellar support team as has grown to support MySQL since we added service for it in 2003. Our team can augment a company’s operational DBAs or can help a small installation move from a developer managed database to a more scalable 24×7 follow the sun operational model. We provide operational remote DBA support to organizations from the startup to divisions of Fortune 500 companies. To learn more, please check out the description of our Remote DBA Services.

Blue Gecko has additionally worked with Patrick Galbraith to contribute DBD::drizzle and Drizzle AMIs for use at Amazon AWS to the Drizzle project. In our core service space, we have worked with the Drizzle team offering infrastructure support to their project for the last two years and are excited to be included in announcement. Many congratulations to their team for making this new micro-kernel database a reality.

updated: Patrick Galbraith will be attending the Drizzle Developer Day on Friday April 15th.

Narada 0.202 (plus AMI!) released

pgalbraith — Thu, 07 Apr 2011 14:40:00 +0000

I’m pleased to announce the release of the Narada Search Engine Application, version 0.202, as well as a ready-to-use AMI, ami-140df07d.

The source release can be found at Launchpad – lp:narada and and a packaged version at http://patg.net/downloads/narada-0.202.tar.gz

What is Narada? Narada is a search engine application. It does have a web application component, but it also has a very novel approach on the back-end in how it implements various functionalities by using Gearman, memcached, Sphinx and either MySQL or Drizzle as a data store. It is somewhat of a proof-of-concept that was borne from an idea I had one late night while working on my first book, Developing Web Applications with Apache, MySQL, memcached, and Perl. I wanted to show how useful using all these great new tools– Gearman, memcached and Sphinx — for scaling out functionality and data, along with MySQL, and later Drizzle.

I realized upon developing this application, not only being an interesting concept to discuss and detail in a book, that it was also a very practical application that others would find very useful, hence I decided to release this project as an open source project that others could put to use and hopefully contribute to.

After initially releasing Narada, there was a PHP added by Eric Day and Java port by the Trond Norbye and the Sun Cloud Computing team that was presented at Java One two years ago. These PHP port is not as up to date as the Perl port and the Java port was lost in the shuffle of mergers, but I intend to at least bring the PHP API up to date soon. One of the ideas I wish to stress is that with Narada, you are not forced to use any particular language for implementation. The workers and web UI can be implemented any way you like. It’s the artichitectural concept of Narada — using Gearman workers, a distributed index like Sphinx, and a write-through cache with memcached, as well as using either MySQL or Drizzle for data store, that is important. All language implementations are included with Narada.

In terms of specific technical functionality, Narada consists of a very simple web application front end with a back end that uses Gearman (Gearman workers specifically) , Sphinx, memcached and MySQL or Drizzle to implement the real functionality of the application. The basic user experience for Narada is that there is what I call a URL queue web UI where one enters URLs that could be thought of as a “seed” URLs.

I thought it was be a great idea to use gearmand to dispatch what I call “url_fetcher” workers that are used obtain content from the remote document of the submitted URL, look for any links and in turn use recursion, to a given depth, fetching yet more remote documents a certain number of times. For each remote document that the url_fetcher worker retrieves, it caches the content of that document temporarily in memcached as well as updates a a very simple ID table in the database that contains a “catalogue” of temporarily cached documents as well as updating a simple counter item in memcached to indicate the number of documents cached. The idea behind this is first for performance so each retrieval doesn’t require insertion of a record containing a blob field into the database, and also an attempt to inevitably “bulk” insert these cached documents at a later point in time, albeit short period of time.

For each document the url_fetcher worker retrieves and caches, it in turn requests another worker through gearmand, the “url_store” worker. The url_store worker, whenever called, checks a the counter item that the url_fetcher worker incremented for each document retrieved and the number checked for to a given value, if matching, determines that the cached documents should be stored in the permanently in the database. These cached documents are bulk-inserted into the database for the benefit of write performance and a simple counter item set to the value of the number of documents stored in the database is cached memcached .

The url_store worker also requests through gearmand an Sphinx reindex worker that when called checks the counter set by the url_store worker for the number of documents stored and if the value matches, the Sphinx indexer is called to rebuild the indexes. The other component to make use of this indexes content is a web UI that allows a user to enter a search term and returns the URLs and an excerpt with the search term highlighted which the user can then navigate to the remote document/site which contains the search term. This functionality is implemented by using Sphinx to return the document ids from that search term to obtain the document content from MySQL and then finally displayed. Sphinx also is used to build the excerpts for the search results displayed.

The image above shows in a diagram the components described in this post. You’ll see that there are several components, all of which are quite scalable making Narada an ideal Search application with scaling built in from conception.

Narada Search results page

The next image shown is an example of the Search results page.

Narada URL Queue page

And finally, an image displaying the URL Queue page, where you enter “seed” URLs.

This release that I have made, 0.202, is mainly changes to the web UI mod_perl handlers, workers as well as core library. Also changed is how I want to develop a language agnostic Sphinx configuration file using YAML, which I have implemented for the Perl port and soon the PHP port. I’ve also made the Perl port as easy to install as possible – with PHP’s deployment model in mind. I’ve also added a lot of documentation about how to install and run Narada.

The AMI I also released, ami-140df07d, is an out-of-the-box ready-to-use AMI that has everything set up for you to immediately run Narada on your website. You will find upon logging in as the “narada” user to your RPM that there is a file in the home directory called “README.1st”. Read this, and you will be on your way to running Narada!

As a final note, next week at the MySQL Users Conference I will be giving a talk and demonstration of Narada. I hope to see you there!

Blue Gecko Drizzle i386 AMI Now Available!

pgalbraith — Wed, 06 Apr 2011 03:06:34 +0000

I’m pleased to announce that we have just released 32-bit Drizzle AMIs, based off of Amazon’s stock 32-bit RPM-based AMIs. Launch an instance of this AMI (and you can use a small instance), log in as “ec2-user”, and you will have an instant Drizzle server to utilize!

This server is utilizing file-based authentication, but you can install any one of the authentication mechanisms by installing the appropriate RPM of your choice found in the home directory of ec2-user in “RPMS”.

The AMI ID is ami-a80df0c1.

Enjoy using this AMI and please let us know if you have any questions, issues or suggestions!

Blue Gecko Drizzle AMIs now available!

pgalbraith — Fri, 18 Mar 2011 15:44:27 +0000

I’m pleased to announce that we’ve made available two Drizzle Amazon AMIs for the community to take advantage of! The AMIs are:

ami-d408fabd – Blue Gecko Drizzle AMI

This AMI is a ready-to-use AMI that comes with a complete Drizzle version 2011.03.13 GA installation as well as the client library and Perl DBD::drizzle pre-installed. Simply launch this AMI and have Drizzle ready to use and explore this great new RDBMS!

ami-9a30c3f3 – Blue Gecko Drizzle Development AMI

This AMI is a development AMI that has all the necessary libraries as well as a complete development environment for development of Drizzle. This AMI does *not* have Drizzle pre-installed. We leave it up to the user of this AMI to install whatever version of Drizzle they want as well as develop new features for drizzle or work on any of the client programs for Drizzle!

We hope you enjoy using these AMIs. Drizzle is a very exciting and promising RDBMS that takes a whole new approach to database development. We intend to keep releasing new AMIs with newer versions of Drizzle and with more client libraries and utilities pre-installed in the future, so stay tuned!

DBD::drizzle 0.303 Released

pgalbraith — Fri, 25 Feb 2011 20:09:21 +0000

I’m pleased to announce the release of DBD::drizzle 0.303! As Drizzle progresses in development, so too does DBD::drizzle. The main issue is that there are changes that I need to keep up to and that’s what this release primarily consists of. Most of the test suite was broken due to an error “Transactional DDL not supported” when “DROP TABLE” was run in the tests. The fix for this was to turn on AutoCommit for the database handle. The other issue was in the table_info() method which due to some changes in Drizzle’s data_dictionary schema (yes, they have a data dictionary!) some values table_info() expected to be set had changed.

So, go forth and get the code!

http://search.cpan.org/~capttofu/DBD-drizzle-0.303/lib/DBD/drizzle.pm

Also:

bzr branch lp:drizzle

DBD::drizzle .301 released

pgalbraith — Fri, 22 Oct 2010 14:57:42 +0000

Dear Perl and Drizzle developers,

I’m pleased to announce the release of DBD::drizzle .301! This release implements several fixes:

* Eric Day’s fix to a bug where a result can be NULL after error.
* A lot of reworking of column_info() to utilize Drizzle’s DATA_DICTIONARY and INFORMATION_SCHEMA as opposed to describe and show commands as well as fixing the catalog test to work with these changes
* Other fixes from DBD::mysql

You can find the source code at Launchpad.net:

lp:dbd-drizzle

As well as CPAN:

http://search.cpan.org/~capttofu/DBD-drizzle-0.301/lib/DBD/drizzle.pm

Please feel free to report bugs to http://rt.cpan.org, and as always, thank you for your interest in DBD::drizzle!