***** TORNADO BACK END ***** Copyright (c) 1999-2010 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com 1.866.NNTP.119 ------------------------------------------------------------------------------- Tornado Backend 1.3.2.678 - Fixed a bug in the HFD caching mechanism that was preventing the use of the cache for more than the first 16K article references. To fully take advantage of the fix, add HeaderFeedCacheSize # to the conf file, where # is 16M. That will cache up to 16M article references in RAM and will reduce disk hits to the filesystem holding the .hfd file. - Improved PostFilter behavior. Now when the PostFilter decides it needs to try again because of an unresponsive PostFilter program, or a PostFilter program handling data incorrectly the PostFilter program will be restarted. In the process the current PostFilter program will be stopped, killing it if needed, and the new one started. - Reworked how article cancels happen on Stormcellar. There is now a cache for the article message-ids that will be used during article looks ups (if it's in there we act as if it is canceled). Meanwhile, on behind the scenes we do a metered cancel of articles. This allows less hits against the messageIdLookup or the History DB. The two directives that control the cache size and cancel rate are ArticleCancelCacheSize and ArticleCancelPause. ArticleCancelCacheSize , if set, should be a positive integer, and uses "millions of articles" as it's units of measure. ArticleCancelPause uses milliseconds, not seconds, as it's units of measure. Ex: ArticleCancelPause 10 // Will create a pause of 10 milliseconds // between article cancels. ArticleCancelCacheSize 8 // Will create a cancel cache capable // of holding 8 million articles. - Cancel control handling syslogs are chatty and have typos ------------------------------------------------------------------------------- Tornado Backend 1.3.0.636 - A new spool directive of 'Offline ' has been added. If the directive is not found in the definition of spool object, the spool is online by default. If the spool object specifies 'Offline True', then, at startup, a spool object will be defined but put into an offline state. No file operations writes will be preformed on a spool that is offline. - Added periodic checking for the existence of the file offline.HUP in the control directory. When found, a quick parse of the spool objects in the configuration file is preformed. If a change of online vs offline state is found between the definition and the current state of the spool, the state of the spool is changed. - Removed unused Feed directive AllowFilter and Configuration directive DefaultAllowFilter - Incoming feed objects may contain the directive IncomingPortNumber that can be used as an additional discriminator in feed association. ------------------------------------------------------------------------------- Tornado Backend 1.1.2.572 - TBE will no longer start with XrefAction Generate and AutoAddGroups TRUE. To override and allow all groups to be added, use -looseadd (#33031) ------------------------------------------------------------------------------- Tornado Backend 1.1.2.560 - *** NOTE: We have changed some license processing code, which may result in the need for us to regenerate your license key. Please run bin/validate to verify your key is still valid. You can always get a new temporary key at http://www.highwinds-software.com/licensing/3daykey.php - -strict is no longer supported. - Stopped Cancel messages showing up with SpoolClassifier turned on (#429208) - Fixed a race condition with AutoAddGroups that could cause the first few articles of a new group to be lost if they arrive simultanously. (#438055) ------------------------------------------------------------------------------- Tornado Backend 1.1.1.556 - The FilterAPI package names have changed from using a prefix of Openwave:: to a prefix of Highwinds::. This impacts any custom built filters that depend on the FilterAPI. Specifically, the two files FilterAPI.pm and FilterAPI/Manager.pm have changed. ------------------------------------------------------------------------------- Tornado Backend 1.1.1.540 - Added support for refeeding in near term order as managed by Tornado Front-End version 1.2.3.540 or later. - Fixed bug where an article would be unretrievable if the colon in the newsgroups field in the header was followed by a tab. - Switched default of stats.weights to be disabled instead of enabled ------------------------------------------------------------------------------- Tornado Backend 1.1.1.490 In header_queue.control, the new depth of the an FE can be specified in terms of another FE. For example, if you want FrontEnd20 to be in the same position as FrontEnd30, you would add the following to the header_queue.control: FrontEnd20 fe:FrontEnd30 The fe: before FrontEnd30 helps distinguish between a frontend name and a numberic position. Still supported are: FrontEnd20 max FrontEnd20 10000 ------------------------------------------------------------------------------- Tornado Backend 1.1.1 New configuration file settings: HeaderFeedQueueCacheSize <#> where <#> is the number of article header references to cache in RAM. Each reference is 28 bytes. Default is 16K references. Max is 16M references. HeaderFeedDiskCacheSize <#> where <#> is the number of article header references cached in RAM before an actual write to disk occurs. Default is 16K references. Max is 256K ------------------------------------------------------------------------------- Tornado Backend 1.1.0 New features & files ----- A new queuing mechanism has been built for the BE to feed article headers to each FE. This mechanism fixes problems where FEs could fail to receive articles or FEs would erroneously receive duplicates of very old articles. The new mechanism creates a new queue file named tornadobe.hfd. It also creates one new file per FE named with the feed name and with the extension .hfp, which stands for Header Feed Position. To convert from your existing queue data files, an upgrade procedure is defined below. The size of the HFD file is determined using the existing value for the size of the dummy feed files. You may have specified the size of the dummy feed with DummyFrontEndQueueSize or with MasterFrontEndQueueSize. Whichever is set is the one used when the HFD file is created. The value is in KB, with a maximum size of 4000000000 KB (4TB, 125 billion article references). However, do not change this value until after the upgrade procedure has been completed. When the upgrade procedure is executed, the HFD will be created using the existing path for the dummy queues and using the existing queue size. Anything in the dummy queues will be copied to the new HFD queue. Be sure to check available disk space when running the upgrade tool. If necessary, the df.1/df.2 files can be copied to a different filesystem and then sym-linked to from their original locations. The HFD can be expanded at any time simply by increasing the size specified in the conf file. When restarted, TornadoBE/Stormcellar will expand the HFD file to the given size and the additional space will start to be used when the previously existing bytes in the HFD file are filled. NOTE: The HFD file is not a sparse file. On startup, the full size specified in the config is taken. New hfp files will be created in the directory specified in each Feed's TornadoQueuePath setting in the feeds.conf file. When a new FE feed is configured, on startup the TBE will place that feed as far back as possible in the queue so that the new FE will receive the maximum history available from the TBE. When a feed is upgraded using the procedure defined below, the feed is placed at the head of the queue as if it already has everything the TBE has to offer. Spool files, active files, MID files, H files, track files and Overview files are unaffected by this upgrade. Also, no FE changes are required. Upgrade procedure ----- Note: The upgrade tool uses the existing TBE config files, which may contain paths relative to the run location of the TBE, which is generally in the tornado install's bin/ dir. The upgrade tool must be run from the same directory that the TBE runs. STOP the TBE No configuration changes should be made Make sure you have enough space on your filesystem(s) for the new data files. The new HFD data file will be the size specified for the current df.1/df.2 data files, so the single HFD data file will be twice the size of one of the df data files. Execute the upgradeToHFD utility from the bin directory where the TBE executable is located If the upgradeToHFD utility reports that it worked, move your current TBE binary out of the way and put the new TBE binary in place. Start the new TBE All FE queues are positioned at the front of the HFD queue. If you need to roll any back, this can be done manually. See below for details. Once upgrading is complete and you are satisified with the performance, df.1 & df.2 can be deleted Downgrade procedure ----- Note: The downgrade tool uses the existing TBE config files, which may contain paths relative to the run location of the TBE, which is generally in the tornado install's bin/ dir. The downgrade tool must be run from the same directory that the TBE runs. STOP the TBE If you increased the size of the HFD beyond 4GB, you will need to change your config file so that DummyFrontEndQueueSize/MasterFrontEndQueueSize is less than 4GB. Make sure you have enough space on your filesystem(s) for the df.1/df.2 files to be created. Execute the downgradeFromHFD utility from the bin directory where the TBE executable is located If the downgradeFromHFD utility reports that it worked, move your new TBE binary out of the way and restore the previous TBE binary. Start the TBE Note that no more than 70M article references can be copied from the HFD into the dummy queues as the dummy queues have a maximum of 35M articles each. All FE queues are positioned at the front of the df.1 queue. If you need to roll any back, this can be done manually. See below for details. Once downgrading is complete and you are satisified with the performance, the .hfd file and the .hfp files can be deleted How to manually adjust FE depth ---- The file header_queue.control can be updated at run time with instructions on how to alter any FE's queue position. Format of file: # comment incomingFeedName can be a number up to 10 digits in length or may say "max" Examples: mrhowell 50 skipper max # Here is a comment The file should be created in the same directory as the active files. Once you have header_queue.control as you like it, execute the bundled script "reload" to cause the BE to read the file and process the requests. The log will show progress with LOG_INFO messages and errors with LOG_ERR messages. Once the commands in header_queue.control are executed, header_queue.control will be deleted. ------------------------------------------------------------------------------- Tornado Backend 1.0.5.448 - Fixed memory leak when article headers are queued - Added support for friendly Diablo retention ------------------------------------------------------------------------------- Tornado Backend 1.0.5.424 - Added commamnd line argument to tornado to suppress writing of $/log/stats.weights ------------------------------------------------------------------------------- Tornado Backend 1.0.5.395 - Added XBEREPLACE - allows for the reconstruction of a spool by number. ------------------------------------------------------------------------------- Tornado Backend 1.0.5.394 - Added deeper retention to history. ------------------------------------------------------------------------------- Tornado Backend 1.0.5.390 - Added Terabyte directive for spool sizes. ------------------------------------------------------------------------------- Tornado Backend 1.0.4.380 - Semantical fix with xover, xover # was interpreted as #- that has been corrected - Eliminate/reduce article loss during server shutdown. ------------------------------------------------------------------------------- Tornado Backend 1.0.4.371 - XOVERs are now correctly bandwidth-limited for MaxBytesPerSecond and FeedMaxBytesPerSecond. - The -strict option will prevent users who send bad commands from being disconnected. - More robust handling of huge overview lines. - Bug fix: Spools larger than 32Gb are now handled correctly when starting with -nocreate. - Added -veritas flag which enables VXFS Direct I/O if -direct is also set. ------------------------------------------------------------------------------- Tornado Backend 1.0.4.370 - Fixed PID file misspelling. - Dummy feed size increase to 2Gb by default. - New slave queueing architecture saves disk space for front-end queues. - "Terabytes" now recognized in spool objects. -------------------------------------------------------------------------------- 1.0.4 - Added a new directive called XrefMaster. Use of this directive allows only "trusted" host names that perform article number generation to store articles. The directive XrefMaster takes one or more wildcard arguments. At least one of these arguments have to match the host field in the Xref: header field when the server is set to XrefAction Pass mode. A match failure in parse mode forces an article rejection with code 0x. This directive would apply to cyclone.conf, typhoon.conf, twister.con, and tornado_be.conf. - Fix a random memory corruption problem which has the potential to cause core dumps. - Master/Slave Frontend queues added. Under this new paradigm DummyQueuePath and DummyQueueSize have been depricated and replaced with MasterQueuePath and MasterQueueSize. This queue is used as a repository for all front ends. Each additional frontend simply maintains a master block of 16 bytes, instead of copying the entire queue. This system is not backwards compatible, and will update automatically. When the (old) queue empties, it will be replaced with a .slave file and unlinked. This will happen if/when it empties. To force this, momentarily turn off your cyclone, which will allow all queues to flush. NOTE: ONCE THIS HAPPENS THE SERVER CANNOT BE ROLLED BACK TO ANY PREVIOUS VERSION - SendToTornados has been depricated, now the -backfill command-line option is used. This will scour the spools and place all articles into the MasterQueue (defined in tornado_be.conf) for use by all frontends. -------------------------------------------------------------------------------- 1.0.3 - Added Front End stats collection. Stats are shown in stats.frontend file. See the file README.stats for the stats description. - Overview pre-caching added, this allows greater scaling in the number of Front Ends a Back End can support (effectively unlimited for shared spools). The cache is sized with the configuration option: OverviewPreCache # where # is size in Kilobytes (0-100,000) (a size of zero disables it) - Dramatic reduction in RAM footprint - AdaptiveSubscriptionPolicy [none|bool|weighted] specified in config file turns on that type of adaptive storage for all spools - -nonblockingclose option to attempt to recover thread resources when closing reader connections. This is useful when a server is receiving many multiple connect/disconencts. - Support for raw devices. use "AccessMode RAW" in a spool object and specify the raw device as the Path. raw devices must be formatted/labeled with the supplied tools/rawtool - Cascade feeding supported. Defining a cascade feed will cause articles that are just about to expire to be send to a downstream server (stormcellar) for archival. Feed Object format: OutgoingHostName outgoing_hostname OutgoingFeedName cascade_name Port 119 AdaptiveCascade true/false - Multiple cascade feeding supported, new directives in .conf are: NumberOfCascadeStreams # CascadeDepthPercent # - Large spools now supported, Spool sizes may now be expressed in "Megabytes" and "Gigabytes", and be up to 2 Terabytes in size. - Weights are exposed in stats.weights - Stormcellar Cascade server feed implemented, stats are shown in stats.cascade - All XOver lines are now individually validated for correctness - -keyinfo option in validate -------------------------------------------------------------------------------- 1.0.2 - Connect limit demoted from Error to Info (-detailed to see it) - LogOORArticles [true/false] and DiscardOORArticles [true/false] added. This will affect articles in groups of more than 1000 articles this catches mis-numbered articles. - Crashing when syslog > 2G fixed (Linux only) - Empty syslogs no longer emit critical errors - Enhanced logging for syslog messages that displayed (m) for non-Solaris Operating Systems -------------------------------------------------------------------------------- 1.0.1 - -crossplatformspools allows shared-spool support. WARNING- turning this on WILL DESTROY backward compatibility on non-network order machines (linux/FreeBSD) - CIDR code optimized - ChainMessageIDRequests True|False (default true) wether or not message ID requests will be passed back to a chaining host - Chaining timeouts loosened up for loaded backends. - -nohistory flag added to trust all backend content - -strict flag for strict/narrow NNTP interpretation - cat.cat file corrupt bug fixed - large spool supported -------------------------------------------------------------------------------- 1.0.0 - Added DummyFrontendQueuePath/DummyFrontendQueueSize so back-end can establish a dummy-feed to copy into new front-ends - Added TornadoAccessMode to config. can not explicitly set how front-ends will access the shared spools - Fixed upstream cancel propogation - Propogate moderated active file list - Added SharedSpools to feed - Added TornadoPath to config -------------------------------------------------------------------------------- Copyright (c) 1999-2010 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com