How-To Subscribe to Data Series Created and Maintained by the JSOC-SDP

A NetDRMS site wishing to mirror (obtain and maintain a copy of) a DRMS data series from another site may subscribe to that series. Such series must first be published by the serving site. (see Series Publication Notes) To subscribe to a data series, a privileged operator runs the subscribe_series client script to request a subscription. This script, which can be run as many times as desired, allows the operator to subscribe to one or more series at a time. The first time it is run, the server creates a directory for the site into which site-specific slony logs will reside. These logs will contain statements germane only to the set of series to which the site is subscribed.

In brief, the server satisfies a client request by sending to the client a file containing a SQL dump of the main series table to which the client is subscribing. The file will also contain other set-up commands depending on the status of the client as a new subscriber or not. For example, if a client requests a subscription to hmi.M_45s, then the dump file will contain commands to create the hmi namespace (if it does not already exist), the hmi.m_45s table (it must not exist at subscription time), the appropriate entries in the hmi.drms_keyword, hmi.drms_segment, hmi.drms_link, and hmi.drms_series tables, and many insert commands. The insert commands will populate the hmi.m_45s table with a copy of all the records from the original corresponding table in the server database at the time of subscription.

After the initial subscription completes (subscribe_series has completed successfully), the client has a static snapshot of the series at the time the command was issued. Changes to the original series on the server will not automatically be applied to the client's database. However, the server will regularly generate site-specific slony logs for the client. These logs contain insert (and delete) statements that, when applied by the client, will update the subscribed series tables so that, in effect, the client tables are kept synchronized with the corresponding server tables. The client must regularly download these log files and apply them to remain synchronized with the server.

It is also possible to discontinue one or more subscriptions with the subscribe_series script. Site-specific logs generated after the completion of subscribe_series in this mode will no longer contain any statements relevant to the unsubscribed series. The un-subscription process does not remove from the client database the data for the series being removed. The series remain intact and function normally in every way; but they will become static snapshots of the server series at the time of un-subscription. Subsequent changes to the corresponding server will not propagate to the client.

Before a client can run subscribe_series, several steps must be taken. First, the client must set up an ssh-agent session so that noninteractive ssh-based communications between the client and server can be established. To do this, follow the directions in SSHKeyNotes, ensuring that you do create a passphrase. Second, the client must edit a configuration file that contains server and client information, such as the server's host machine and port, the server account that ssh uses, the path to the client's scp command, client directories to receive the slony logs, etc. We provide a documented template configuration file at base/drms/replication/etc/repclient.template.cfg (see the section entitled Configuration File below for more information). This configuration file is used for all client-side programs/scripts, including subscribe_series. Finally, the client must create a “list” file that lists series and the action to take on these series (e.g. subscribe or unsubscribe). We provide another documented template configuration file at base/drms/replication/etc/subscribe_list.template.cfg. In short, the list file contains two white-space separated columns. The first contains series names, and the second contains the text subscribe or unsubscribe. subscribe will cause the client to subscribe to the named series; unsubscribe will remove the named series from the set of subscribed series.

After completing the set-up, the client operator runs subscribe_series by providing three arguments: the path to the configuration file (Configuration File), the path to the list file, and the path to the ssh-agent source file. A flag -r may also be applied in the event there was a recoverable error from a previous run of subscribe_series. If the -r flag is not provided, then all state files will be wiped clean before subscribe_series is run. If it is provided, then subscribe_series will attempt to continue from where it last left off. (A subscription process might be interrupted by a time-out or outage on the server, a power failure, or a client-side problem that has since been corrected.) The subscription process will complete with a message denoting success or failure. Please see the section entited Troubleshooting for information about troubleshooting subscription failures.

Configuration File

The configuration file contains configuration information needed by several client-side programs and scripts (subscribe_series, get_slony_logs.pl). It consists of several key-value pairs separated by commas (values appropriate when the Stanford JSOC is the server are included in parentheses):

node The name of the subscription client. Must be globally unique across all NetDRMS sites. This string will be used in various state files and in file/directory names. kRSServer The full domain name of the server (solarport.stanford.edu) kRSUser The account on kRSServer that will be used for data transfer (jsocexp) kRSTriggerDir The directory accessible from the kRSUser account where data files will be staged (/data/pgsql/slon_logs/live/triggers/) kRSPort The port on kRSServer that will be used for data transfer (55000) pg_host The client machine that hosts the client PostgreSQL database that will contain the replicated data series pg_port The port on the pg_host machine that will be used for communication with the data-series database pg_user The PostgreSQL user that will own the replicated series (slony) pg_dbname The name of the PostgreSQL database that resides on pg_host slony_cluster The name of the Slony cluster to which this node belongs (jsoc) kLocalLogDir The client directory that will contain log files kLocalWorkingDir The client directory that will contain temporary working files kSQLIngestionProgram The path to the script/program that will ingest the site-specific slony logs — usually the path to get_slony_logs.pl kDeleteSeriesProgram The path to the program delete_series used to delete DRMS data series on the client attempts The number of attempts that the client should make when looking for the flag file indicating the sql file is ready on the server ingestion_path The local directory that will contain the ingestion "die" file — used by get_slony_logs.pl scp_cmd The absolute path to the client's scp program. ssh_cmd The absolute path to the client's ssh program. rmt_slony_dir The absolute path, accessible from the kRSUser account, on the server to the directory that contains the site-specific slony logs (/data/pgsql/slon_logs/live/site_logs) slony_logs The client directory that will contain the downloaded site-specific slony logs PSQL The path to the client's psql program, and any flags needed to run psql as the pg_user user, like -h pg_host email_list The email account to which error messages will be sent

Troubleshooting

There are a number of problems that could occur during the subscription process. The client might not have a "clean" environment; there might be an out-of-date _jsoc namespace (used internally by Slony); the series being subscribed to may exist in some form already; there might be a bad parameter in the configuration file; or there might be directory permission issues, etc. The server might also have an unclean environment. Often times it is easiest to clean up both environments and try again as a new subscriber (please see the section entitled Cleaning Up below).

Cleaning Up

To clean the subscription client environment: 

DELETE FROM admin.ns WHERE name = '<schema>';

To clean the subscription server environment:

Re-subscribing

You might occasionally need to re-subscribe to a series. There could be bugs at the JSOC or in your site's NetDRMS that result in your site missing some DRMS records (or having extra records). An easy way to reconcile these differences is to unsubscribe from the series, then to re-subscribe to the series. If you unsubscribe from a series by running the appropriate subscribe_series command: subscribe_series <path>/repclient.cfg <path>/subscribe_list.cfg <path>/.ssh-agent_rs where subscribe_list.cfg contains this line: hmi.someseries                    unsubscribe and you answer 'Y' to this prompt: Would you like to delete series hmi.someseries? (Y/N) then subscribe_series will delete all SUMS Storage Units (SUs) that you have ingested into your SUMS. subscribe_series deletes the series and all associated SUs by calling a program, delete_series, that is part of your NetDRMS. So, if your goal is to keep the SUs, but unsubscribe to a series, do not use this method as-is.

There are three methods that can be used to unsubscribe a series without deleting the SUs until I can add a new options that causes SUs to be preserved:

  1. Manually edit your subscribe_series script. In the DeleteSeries() function, modify the following line: 

echo "yes"$'\n'"yes"$'\n' | "$delseriesprog" "$series" JSOC_DBUSER=slony

to 

echo "yes"$'\n'"yes"$'\n' | "$delseriesprog" "-k" "$series" JSOC_DBUSER=slony

Add a "-k" (lower-case k) to the command line to the program delete_series. After you do this, then you can run subscribe_series with an unsubscribe line in the subscribe_list.cfg file. Remember to revert your subscribe_series program back to the way it was when you have finished unsubscribing.

  1. Run subscribe_series with an unsubscribe parameter in the subscribe_list.cfg file as you normally would, BUT answer "N" to this prompt: 

Would you like to delete series hmi.someseries? (Y/N)

Then after the unsubscription process has completed, run this program: 

> delete_series -k hmi.someseries  JSOC_DBUSER=slony
  1. Run the delete_series command ahead of time. There is a bit of a timing issue involved, so this method may not work if you happen to run it at just the wrong time. But recovering from an unlucky timing is easy. Run these commands in order: 

> get_slony_logs.pl <path>/repclient.cfg # This is probably running via a cron job - force a run of it.
> delete_series -k hmi.someseries  JSOC_DBUSER=slony
> subscribe_series repclient.cfg subscribe_list.cfg <path>/.ssh-agent_rs # an unsubscribe command for hmi.someseries is in the subscribe_list.cfg file

If a new Slony log gets generated for you after you run get_slony_logs.pl, but before your subscribe_series command completes, then you may receive a Slony log with an insert statement for the series you just deleted, and then your next get_slony_logs.pl command would fail. The chances of this happening are small, but not zero.

At this point, you will have successfully unsubscribed from hmi.someseries, but you will still have the SUs associated with that series in your SUMS. The next step is to subscribe to the series you just unsubscribed from: 

> subscribe_series <path>/repclient.cfg <path>/subscribe_list.cfg <path>/.ssh-agent_rs

where subscribe_list.cfg contains this line: 

hmi.someseries                    subscribe

Finally, if you use the JMD, you will need to set-up the hmi.someseries database table so that the JMD knows to fetch SUs for newly ingested records. To do that, run: 

> sunum_queue_trigger_sampled.pl <arguments>

Unfortunately, I have never run this program so I do not which arguments should be provided, but the good news is that since you had the JMD fetching SUs for hmi.someseries before the unsubscription occurred, you ran sunum_queue_trigger_sampled.pl before. Use the same command you did the last time you subscribed to hmi.someseries.