***** STORMCELLAR ***** Copyright (c) 1999-2010 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com 1-866-NNTP-119 866-668-7119 ------------------------------------------------------------------------------- Stormcellar 1.3.5.678 - Reworked how article cancels happen on Stormcellar. There is now a cache for the article message-ids that will be used during article looks ups (if it's in there we act as if it is canceled). Meanwhile, on behind the scenes we do a metered cancel of articles. This allows less hits against the messageIdLookup or the History DB. The two directives that control the cache size and cancel rate are ArticleCancelCacheSize and ArticleCancelPause. ArticleCancelCacheSize , if set, should be a positive integer, and uses "millions of articles" as it's units of measure. ArticleCancelPause uses milliseconds, not seconds, as it's units of measure. Ex: ArticleCancelPause 10 // Will create a pause of 10 milliseconds // between article cancels. ArticleCancelCacheSize 8 // Will create a cancel cache capable // of holding 8 million articles. - Improved PostFilter behavior. Now when the PostFilter decides it needs to try again because of an unresponsive PostFilter program, or a PostFilter program handling data incorrectly the PostFilter program will be restarted. In the process the current PostFilter program will be stopped, killing it if needed, and the new one started. - Added support for unnumbered articles by allowing a mixture of numbered and unnumbered articles on the same stormcellar. Only numbered articles' headers will be queued for delivery to TornadoFEs. - Fixed a bug in the HFD caching mechanism that was preventing the use of the cache for more than the first 16K article references. To fully take advantage of the fix, add HeaderFeedCacheSize # to the conf file, where # is 16M. That will cache up to 16M article references in RAM and will reduce disk hits to the filesystem holding the .hfd file. - Reworked cancel control SYSLOGs that were chatty and had typos. ------------------------------------------------------------------------------- Stormcellar 1.3.0.636 - A new spool directive of 'Offline ' has been added. If the directive is not found in the definition of spool object, the spool is online by default. If the spool object specifies 'Offline True', then, at startup, a spool object will be defined but put into an offline state. No file operations writes will be preformed on a spool that is offline. - Added periodic checking for the existence of the file offline.HUP in the control directory. When found, a quick parse of the spool objects in the configuration file is preformed. If a change of online vs offline state is found between the definition and the current state of the spool, the state of the spool is changed. - Removed unused Feed directive AllowFilter and Configuration directive DefaultAllowFilter - Incoming feed objects may contain the directive IncomingPortNumber that can be used as an additional discriminator in feed association. ------------------------------------------------------------------------------- StormCellar 1.1.2.560 - *** NOTE: We have changed some license processing code, which may result in the need for us to regenerate your license key. Please run bin/validate to verify your key is still valid. You can always get a new temporary key at http://www.highwinds-software.com/licensing/3daykey.php ------------------------------------------------------------------------------- StormCellar 1.1.1.540 - Added support for refeeding in near term order as managed by Tornado Front-End version 1.2.3.540 or later. - Switched default of stats.weights from enabled to disabled - Fixed bug where an article would be unretrievable if the colon in the newsgroups field in the header was followed by a tab. ------------------------------------------------------------------------------- StormCellar 1.1.1.490 In header_queue.control, the new depth of the an FE can be specified in terms of another FE. For example, if you want FrontEnd20 to be in the same position as FrontEnd30, you would add the following to the header_queue.control: FrontEnd20 fe:FrontEnd30 The fe: before FrontEnd30 helps distinguish between a frontend name and a numberic position. Still supported are: FrontEnd20 max FrontEnd20 10000 ------------------------------------------------------------------------------- StormCellar 1.1.1 New configuration file settings: HeaderFeedQueueCacheSize <#> where <#> is the number of article header references to cache in RAM. Each reference is 28 bytes. Default is 16K references. Max is 16M references. HeaderFeedDiskCacheSize <#> where <#> is the number of article header references cached in RAM before an actual write to disk occurs. Default is 16K references. Max is 256K ------------------------------------------------------------------------------- StormCellar 1.1.0 New features & files ----- A new queuing mechanism has been built for the BE to feed article headers to each FE. This mechanism fixes problems where FEs could fail to receive articles or FEs would erroneously receive duplicates of very old articles. The new mechanism creates a new queue file named tornadobe.hfd. It also creates one new file per FE named with the feed name and with the extension .hfp, which stands for Header Feed Position. To convert from your existing queue data files, an upgrade procedure is defined below. The size of the HFD file is determined using the existing value for the size of the dummy feed files. You may have specified the size of the dummy feed with DummyFrontEndQueueSize or with MasterFrontEndQueueSize. Whichever is set is the one used when the HFD file is created. The value is in KB, with a maximum size of 4000000000 KB (4TB, 125 billion article references). However, do not change this value until after the upgrade procedure has been completed. When the upgrade procedure is executed, the HFD will be created using the existing path for the dummy queues and using the existing queue size. Anything in the dummy queues will be copied to the new HFD queue. Be sure to check available disk space when running the upgrade tool. If necessary, the df.1/df.2 files can be copied to a different filesystem and then sym-linked to from their original locations. The HFD can be expanded at any time simply by increasing the size specified in the conf file. When restarted, TornadoBE/Stormcellar will expand the HFD file to the given size and the additional space will start to be used when the previously existing bytes in the HFD file are filled. NOTE: The HFD file is not a sparse file. On startup, the full size specified in the config is taken. New hfp files will be created in the directory specified in each Feed's TornadoQueuePath setting in the feeds.conf file. When a new FE feed is configured, on startup the TBE will place that feed as far back as possible in the queue so that the new FE will receive the maximum history available from the TBE. When a feed is upgraded using the procedure defined below, the feed is placed at the head of the queue as if it already has everything the TBE has to offer. Spool files, active files, MID files, H files, track files and Overview files are unaffected by this upgrade. Also, no FE changes are required. Upgrade procedure ----- Note: The upgrade tool uses the existing TBE config files, which may contain paths relative to the run location of the TBE, which is generally in the tornado install's bin/ dir. The upgrade tool must be run from the same directory that the TBE runs. STOP the TBE No configuration changes should be made Make sure you have enough space on your filesystem(s) for the new data files. The new HFD data file will be the size specified for the current df.1/df.2 data files, so the single HFD data file will be twice the size of one of the df data files. Execute the upgradeToHFD utility from the bin directory where the TBE executable is located If the upgradeToHFD utility reports that it worked, move your current TBE binary out of the way and put the new TBE binary in place. Start the new TBE All FE queues are positioned at the front of the HFD queue. If you need to roll any back, this can be done manually. See below for details. Once upgrading is complete and you are satisified with the performance, df.1 & df.2 can be deleted Downgrade procedure ----- Note: The downgrade tool uses the existing TBE config files, which may contain paths relative to the run location of the TBE, which is generally in the tornado install's bin/ dir. The downgrade tool must be run from the same directory that the TBE runs. STOP the TBE If you increased the size of the HFD beyond 4GB, you will need to change your config file so that DummyFrontEndQueueSize/MasterFrontEndQueueSize is less than 4GB. Make sure you have enough space on your filesystem(s) for the df.1/df.2 files to be created. Execute the downgradeFromHFD utility from the bin directory where the TBE executable is located If the downgradeFromHFD utility reports that it worked, move your new TBE binary out of the way and restore the previous TBE binary. Start the TBE Note that no more than 70M article references can be copied from the HFD into the dummy queues as the dummy queues have a maximum of 35M articles each. All FE queues are positioned at the front of the df.1 queue. If you need to roll any back, this can be done manually. See below for details. Once downgrading is complete and you are satisified with the performance, the .hfd file and the .hfp files can be deleted How to manually adjust FE depth ---- The file header_queue.control can be updated at run time with instructions on how to alter any FE's queue position. Format of file: # comment incomingFeedName can be a number up to 10 digits in length or may say "max" Examples: mrhowell 50 skipper max # Here is a comment The file should be created in the same directory as the active files. Once you have header_queue.control as you like it, execute the bundled script "reload" to cause the BE to read the file and process the requests. The log will show progress with LOG_INFO messages and errors with LOG_ERR messages. Once the commands in header_queue.control are executed, header_queue.control will be deleted. ------------------------------------------------------------------------------- StormCellar 1.0.2.448 - Fixed memory leak when article headers were queued - Added support for friendly Diablo retention ------------------------------------------------------------------------------- StormCellar 1.0.1.384 - Added deeper retention to history. ------------------------------------------------------------------------------- StormCellar 1.0.1.380 - Eliminate/reduce article loss during server shutdown. ------------------------------------------------------------------------------- StormCellar 1.0.1.371 - More robust handling of huge overview lines. - Bug fix: Spools larger than 32Gb are now handled correctly when starting with -nocreate. - Added -veritas flag which enables VXFS Direct I/O if -direct is also set. ------------------------------------------------------------------------------- StormCellar 1.0.1.370 - Error message cleanups. - Dummy feed size increased to 2Gb by default. - New slave queueing architecture saves disk space for front-end queues. - "Terabytes" now recognized in spool objects. ------------------------------------------------------------------------------- 1.0.1 - Fix a random memory corruption problem which has the potential to cause core dumps. -------------------------------------------------------------------------------- 1.0.0 - Stormcellar created -------------------------------------------------------------------------------- Copyright (c) 1999-2010 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com 1-866-NNTP-119 866-668-7119