There are two primary pieces to using plugins. The first is getting the data, and the second is using the data.

Getting the data

To create a plugin, a module only has to execute ctools_get_plugins with the right data:

  ctools_include('plugins');
  ctools_get_plugins($module, $type, [$id])

In the above example, $module should be your module's name and $type is the type of the plugin. It is typically best practice to provide some kind of wrapper function to make this easier. For example, Panels provides the following functions to implement the 'content_types' plugin:

/**
 * Fetch metadata on a specific content_type plugin.
 *
 * @param $content type
 *   Name of a panel content type.
 *
 * @return
 *   An array with information about the requested panel content type.
 */
function panels_get_content_type($content_type) {
  ctools_include('context');
  ctools_include('plugins');
  return ctools_get_plugins('panels', 'content_types', $content_type);
}

/**
 * Fetch metadata for all content_type plugins.
 *
 * @return
 *   An array of arrays with information about all available panel content types.
 */
function panels_get_content_types() {
  ctools_include('context');
  ctools_include('plugins');
  return ctools_get_plugins('panels', 'content_types');
}

As a plugin creator, your module can also implement a hook to give more information about this plugin, and to enable a few features that are not normally enabled. If you need any of these features, simply implement hook_ctools_plugin_TYPE (where TYPE is the same $type sent to ctools_get_plugins). This isn't a true hook, it will only be called for the $module that was given. This hook returns an array:

/**
 * Inform CTools that the layout plugin can be loaded from themes.
 */
function panels_ctools_plugin_layouts() {
  return array(
    'load themes' => TRUE,
  );
}

The following information can be specified:

cache: If set to TRUE, the results of ctools_get_plugins will be cached in the 'cache' table, thus preventing .inc files from being loaded. ctools_get_plugins looking for a specific plugin will always load the appropriate .inc file.
defaults: An array of defaults that should be applied to each plugin; this can be used to ensure that every plugin has the basic data necessary. These defaults will not ovewrite data supplied by the plugin. This could also be a function name, in which case the callback will be used to provide defaults.
load themes: If set to TRUE, then plugins can be supplied by themes as well as modules. If this is the case, all themes that are currently enabled will provide a plugin: NOTE: Due to a slight UI bug in Drupal, it is possible for the default theme to be active but not enabled. If this is the case, that theme will NOT provide plugins, so if you are using this feature, be sure to document that issue. Also, themes set via $custom_theme do not necessarily need to be enabled, but the system has no way of knowing what those themes are, so the enabled flag is the only true method of identifying which themes can provide layouts.
hook: The name of the hook used to collect data for this plugin. Normally this is $module . '_' . $type -- but this can be changed here. If you change this, you MUST be sure to document this for your plugin implementors as it will change the format of the specially named hook.
process: An optional function callback to use for processing a plugin. This can be used to provide automated settings that require code. The parameters on this callback are: callback(&$plugin, $info) where $plugin is a reference to the plugin as processed and $info is the fully processed result of hook_ctools_plugin_api_info().
info file: If set to TRUE, then the plugin will look for a .info file instead of a .inc. Internally, this will look exactly the same, though obviously a .info file cannot contain functions. This can be good for styles that may not need to contain code.
extension: Can be used to change the extension on a file. By default the extension will be "inc", though it will default to "info" if "info files" is set to true. Do not include the dot in the extension if changing it, that will be added automatically.

In addition, there is a 'module', 'type' and 'hook' settings; these are for internal use of the plugin system and you should not change these.

Using the data

Each plugin returns a packet of data, which is added to with a few defaults. Each plugin is guaranteed to always have the following data:

name: The name of the plugin. This is also the key in the array, of the full list of plugins, and is placed here since that is not always available.
module: The module that supplied the plugin.
file: The actual file containing the plugin.
path: The path to the file containing the plugin. This is useful for using secondary files, such as templates, css files, images, etc, that may come with a plugin.

Any of the above items can be overridden by the plugin itself, though the most likely one to be modified is the 'path'. The most likely data (beyond simple printable data) for a plugin to provide is a callback. The plugin system provides a pair of functions to make it easy and consistent for these callbacks to be used. The first is ctools_plugin_get_function, which requires the full $plugin object.

/**
 * Get a function from a plugin, if it exists. If the plugin is not already
 * loaded, try ctools_plugin_load_function() instead.
 *
 * @param $plugin
 *   The loaded plugin type.
 * @param $callback_name
 *   The identifier of the function. For example, 'settings form'.
 *
 * @return
 *   The actual name of the function to call, or NULL if the function
 *   does not exist.
 */
function ctools_plugin_get_function($plugin, $callback_name)

The second does not require the full $plugin object, and will load it:

/**
 * Load a plugin and get a function name from it, returning success only
 * if the function exists.
 *
 * @param $module
 *   The module that owns the plugin type.
 * @param $type
 *   The type of plugin.
 * @param $id
 *   The id of the specific plugin to load.
 * @param $callback_name
 *   The identifier of the function. For example, 'settings form'.
 *
 * @return
 *   The actual name of the function to call, or NULL if the function
 *   does not exist.
 */
function ctools_plugin_load_function($module, $type, $id, $callback_name) {

Both of these functions will ensure any needed files are included. In fact, it allows each callback to specify alternative include files. The plugin implementation could include code like this:

  'callback_name' => 'actual_name_of_function_here',

Or like this:

  'callback_name' => array(
    'file' => 'filename',
    'path' => 'filepath', // optional, will use plugin path if absent
    'function' => 'actual_name_of_function_here',
  ),

An example, for 'plugin_example' type

$plugin = array(
  'name' => 'my_plugin',
  'module' => 'my_module',
  'example_callback' => array(
    'file' => 'my_plugin.extrafile.inc',
    'function' => 'my_module_my_plugin_example_callback',
  ),
);

To utilize this callback on this plugin:

if ($function = ctools_plugin_get_function($plugin, 'example_callback')) {
  $function($arg1, $arg2, $etc);
}

Document your plugins!

Since the data provided by your plugin tends to be specific to your plugin type, you really need to document what the data returned in the hook in the .inc file will be or nobody will figure it out. Use advanced help and document it there. If every system that utilizes plugins does this, then plugin implementors will quickly learn to expect the documentation to be in the advanced help.