***** TORNADO Superscaling Reader Server ***** Copyright (c) 1999-2008 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com 1.866.NNTP.119 ------------------------------------------------------------------------------- Tornado Front-End 1.1.0.483 - Tightened Message-ID retrieval to follow filtering rules. If there is a per-feed override of any of the Group Access directives including SpecialSubscription, Subscription, FilterSubscription, GroupAllowFile or GroupDenyFile, then retrieval of an article by MID must perform an additional check. Specifically, the xref line of the article is retrieved and each group in the xref is checked against the GroupAccess parameters. If ANY group is disallowed by filter rules on the feed, a '430 Article Not Found' will be returned. Ticket # 438292 - Enhanced AutoAddGroups to add new groups to LIST output immediately. ------------------------------------------------------------------------------- Tornado Front-End 1.0.9.481 - Added WelcomeMessageSuffix feed directive. Allows an addendum of another string to the welcome message that remains static regardless of the Welcome Message override sent from an AuthProgram. Added a DefaultWelcomeMessageSuffix to the tornado.conf-EXAMPLE file. Ticket # 30790 ------------------------------------------------------------------------------- Tornado Front-End 1.0.9.469 - Fixed bug preventing some groups from moving to chunk sizes larger than "N" in hfiles. ------------------------------------------------------------------------------- Tornado Front-End 1.0.9.456 - Improved effectiveness of startup audit code - Improved ofile-size reduction rate - Improved responsiveness to process stop requests - Clarified error logs - We fixed a bug that caused large groups to loose a significant amount of retention. Bug was only exposed by high load and large groups. Ticket #37336 ------------------------------------------------------------------------------- Tornado Front-End 1.0.8.448 - Updating account stats on second chance retrievel. Ticket #232432 ------------------------------------------------------------------------------- Tornado Front-End 1.0.8.447 - Fixed bug where retention pattern was being applied on the LIST command, causing undesirable performance degradation. Ticket #245935 ------------------------------------------------------------------------------- Tornado Front-End 1.0.8.442 -Updated retention pattern to support multiple expressions ------------------------------------------------------------------------------- Tornado Front-End 1.0.7.438 -Added feature to audit the OV files against the active table upon startup. Pass -validateovtestonly on the command line to only log any fixes that would have occurred. ------------------------------------------------------------------------------- Tornado Front-End 1.0.7.424 -Tightened enforcement of retention hours -Requesting too low an aritcle number now returns 423 -Change 'No such article' syslogs so that they are unique -Added DefaultHeaderStreams and HeaderStreams directives -Support for log files >2GB -Performance improvement to BE-->FE header downloads -Changed behavior of HEAD,ARTICLE,BODY and STAT commands to not change current article number -PruneInterval and PruneThreshold values must be added to the .conf file. Pruning is the operation of raising the low bounds of a group to the first article number with valid xover information. Xover information self expires as new xover data is stored over the oldest xover information. This operation is enabled by setting two key/value pairs in the tornado.conf file: PruneInterval and PruneThreshold. The first of these, PruneInterval, specifies the number of minutes between attempts to starting pruning. Starting pruning fails if the previous run is still running. PruneThreshold allows the administrator to determine the density of the xover information required when determining the low bounds of the group. This is specified as a n out of m value or n valid xovers within a window of m article numbers. For example, 1/1 specifies that the pruning algorithm should find the first valid xover and set the new low of a group to that article number. A value of 80/100 specifies a density of 80% or 80 out of 100 valid xovers must be found in a sliding window before the low bounds can be set to the lowest article number of the window. Using any value other than 1/1 ensures that the group does not grow artificially large because of 1 lingering xover where there is a large 'hole' in the article numbers between the first xover and the next valid xover. - Added Backend Proxy feature This feature was added to address a specific problem when using direct spools with additional storage accessible via direct spools on another Tornado frontend machine. This is a middle ground between direct spools and direct connections to a backend and is meant to relieve load on a Tornado backend. Three default keys have been added to tornado.conf: DefaultAllowBackendProxy, DefaultBackendProxyConnectionMin and DefaultBackendProxyConnectionMax. These are required keys. Three corresponding keys are allowed within a feed object: AllowBackendProxy, BackendProxyConnectionMin and BackendProxyConnectionMax. Additionally, BackendProxy is used within a feed object to specify the name and port of the frontend that is proxying a backend. To configure a frontend to use a second frontend as a proxy, set BackendProxy to the : of the second frontend. Additionally, set BackendProxyConnectionMin and BackendProxyConnectionMax to the minimum and maximum number of cached connections that should be maintained between the two frontends To configure a frontend to serve as a proxy, add a feed with 'AllowBackendProxy True'. ------------------------------------------------------------------------------- Tornado Front-End 1.0.7.403 -PruneInterval is now a required parameter in tornado.conf ------------------------------------------------------------------------------- Tornado Front-End 1.0.7.395 -Bug fix: It is no longer possible to second chance around retention hours using a message-ID ------------------------------------------------------------------------------- Tornado Front-End 1.0.7.394 -Added NNTP commands for proxied article retrieval XBEARTICLE XBEBODY XBEHEAD -Added new feed directives AllowBackendProxy ( True | False | Yes | No ) Allows a front end to service XBEARTICLE, XBEBODY and XBEHEAD. Default False. BackendProxy ( Host:Port ) Designates a front end to service proxied article requests BackendProxyConnectionMin Per feed minimum number of connections to a proxy Frontend BackendProxyConnectionMax Per feed maximum number of connections to a proxy Frontend -Added new config directives DefaultAllowBackendProxy (True | False) Flag to allow or denigh servicing of XBE* commands DefaultBackendProxyConnectionMin Default minimum number of connections to a proxy Frontend DefaultBackendProxyConnectionMax Default maximum number of connections to a proxy Frontend MessageIDDepth Specifies the depth of a global midfile MessageIDMultiPath Specifies the path of a global midfile -Bug fix: It is no longer possible to second chance around retention hours ------------------------------------------------------------------------------- TORNADO FRONT-End 1.0.7.390 - Added required PostFilterErrorAction configuration key/value where the value can be either 'Accept' or 'Reject'. This determines what action to take when the external postfilter is unreachable due to either the program terminating or failing to respond to read/writes. Accept was the previous action taken in all cases. Reject causes the user's article posting to fail completely. - Improved latency on ARTICLE command. - Full cancel.control support include cross-post cancellation. - Added capability to allow Terabyte directive for spool sizes. ------------------------------------------------------------------------------- Tornado Front-End 1.0.6.380 - Semantical fix with xover, xover # was interpreted as #- that has been corrected - 3 fields added to .backend stats (see stats readme) - Large o-file support, on all supported architectures o-files may now grow larger than 2G in order to accomodate huge (>10 Mil) groups. This support is enabled through the use of "MaxOVSize #" in the conf file, where '#' is size in Millions of bytes, default 2000. The upper bound of this number is dependant upon your file system. - Eliminate/reduce article loss during server shutdown. - Port2Only added as a feed directive. If unspecified it is false. If set to true then this object may only be invoked on the port specified by the -port2 # command line option - "FastStart" implemented. Articles pulled over TCP/IP from the backend will now begin transmission to the reader immediately and asynchronously. This transmission begins with the first packet to arrive from the backend, so the frontend no longer waits for the entire article to be recieved. The result is a substantial reduction in latency between the time an article is asked for and recieved over TCP, particularly over loaded or high-latency networks. ------------------------------------------------------------------------------- Tornado Front-End 1.0.6.371 - Tornado Front-End now ships with start.conf-EXAMPLE to prevent accidental overwrite of start.conf on upgrade. - XOVERs are now correctly bandwidth-limited for MaxBytesPerSecond and FeedMaxBytesPerSecond. - The -strict option will prevent users who send bad commands from being disconnected. - More robust handling of huge overview lines. - Expensive statistics generation moved out of main thread for more responsive shutdown. - Bug fix: enabled BandwidthLimiting directives for all Tornado Front-Ends. - NOTE BUG FIX: Tornado FE now heeds the AutoAddGroups setting when receiving headers from the backend. Please check your AutoAddGroups setting on upgrade! ------------------------------------------------------------------------------- 1.0.6.370 - Automated cleanup of damaged Xref lines. - Better stability on shutdown. - DefaultMessageIDMultiPath added to tornado.conf ------------------------------------------------------------------------------- 1.0.6 - Feed option "AllowStat [true|false]" default false. If set true then list XSTATREADER, list XSTATBACKEND, list XSTATRETENTION and list XSTATRESET commands will be honored. These commands are analagous to (and use the same internal counters as) the ./statsnow family of stat outputs, with one important exception, the stats must be manually reset. This is in order NOT to interfere with normal operations. XSTATREADER format ( delimited) Source Feed -> Value of IncomingFeedName Xover Requests -> Number of NNTP "XOVER/XHDR/XPAT" requests received Xover Bytes -> Number of Bytes sent in response to "XOVER/XHDR/XPAT" Article Requests -> Number of NNTP "ARTICLE/HEAD/BODY" requests received Article Bytes -> Number of Bytes sent in response to "ARTICLE/HEAD/BODY" List Requests -> Number of NNTP "LIST" requests received List Bytes -> Number of Bytes sent in response to "LIST" Newnews Requests -> Number of NNTP "NEWNEWS" requests received Newnews Bytes -> Number of Bytes sent in response to "NEWNEWS" Post Requests -> Number of NNTP "POST" requests received Post Bytes -> Number of Bytes in articles POST'd. Groups Visited -> Number of Newsgroups visited during statistics interval Active Hosts -> Number of Currently Connected Unique hosts for this feed Active Connections -> Number of Currently Open Connections for this feed Peak Hosts -> Peak number of unique hosts in this interval. Peak Connections -> Peak number of connections in this interval. Auths Attempted -> Number of authentications attempted Auths Successful -> Number of authentications that succeed XSTATBACKEND format ( delimited) Backend Number -> Which number this server has been specified to be in the Fetch Priority -> What fetching priority this server has (1 = highest) Articles Found -> How many articles have been fetched Articles Failed -> How many articles were attempted, but not available Headers Found -> How many headers have been fetched Headers Failed -> How many headers were attempted, but not available TCP Received -> How many bytes have been recieved via TCP ("over the wire") Shared Received -> How many bytes have been pulled from a local mount point (shared spool) Header Bytes -> How many bytes have been fetched for the HEAD command Connection Available -> How many times a free connection was found to retrieve an article from the backend. Highest Connection Count -> How many connections have been opened to the backend Connection Pool -> How many connection can be opened Connection Not Available -> How many times all connections were busy, and a reader had to wait for one to free up (this should always be 0) Backlog -> How many headers are on the backend machine waiting to be sent Name -> Name of the feed Connections -> How many backend connections are active XSTATRESET resets all counts as if a log interval had occured. XSTATRETENTION attempts to find the first 5 numerically contiguous articles, assuming this to be the actual begining of retention. Failing that it will return the age of the first article it encounters. This age is independant of the age the article claims to be, its the age of the article since it was recieved by this frontend, in hours. - Fix a memory leak that can cause exhaustion of all RAM use over long periods of running time. -------------------------------------------------------------------------------- 1.0.5 WARNING: 1.0.5 uses new data structures which are backwards compatible with all previous version, but once run, CANNOT BE ROLLED BACK. - RAW spools allowed as caching spools, note addition of /tools/rawtool to prep raw partitions for use - -fasthangup option added to hang up on people trying too many connections too quickly - Byte scavenging code added for H-files. This will reduce the size of the H-file footprint by up to 30% - MessageIDPath depricated - -prune # depricated and moved to conf file as PruneInterval <# minutes> 0 is disabled, minimum 30. - Added CompletionMaskEnable [true|false] to feed objects, default true - CompletionMaskArticleThreshold <# articles> to conf file. During an xover wait for # contiguous articles before displaying them to the user, default 0 (disabled) - CompletionMaskSubscription/FilterSubscription what groups to apply the CompletionMask to (typically binary groups) - AdaptiveCommit [true|false] in feed objects will only commit articles to local storage based on readership frequency. Defining this still honors CommitPercent, so it is recommended only for use with CommitPercent 100. - -headerstats must be enabled to output stats.headers - IncludeUserNameInXtrace [true|false] config option (default false) - AllowDiag [true|false] now on feeds, with it true, the HEAD command will prepend X-Tornado-Method to show where the article came from - Support -keyinfo option in validate -------------------------------------------------------------------------------- 1.0.4 - Added -prune option minimum 360, this will survey the groups and prune out large gaps. This is particularly useful when there are lots of large backends. - Header stats now includes the backend number - Custom messageID index now checks for previous isntances of articles, so it is only added for the first server that declares it - Multi-threaded message handling -------------------------------------------------------------------------------- 1.0.3 - Connect limit demoted from Error to Info (-detailed to see it) - MessageIDMultiPath added to feed object to distribute messageID lookup - Enhanced logging for syslog messages that displayed (m) for non-Solaris Operating Systems - LogOORArticles and DiscardOORArticles added to config file. This controls the policy for articles that are out of range of a group being added, both default true. - Some error messages downgraded to info - stats.incoming/outgoing have been removed (they were unnused) and replaced with stats.headers which logs a line for each header fed from the back end. See stats.readme for more info - Bogus article numbers filtered to prevent overview damage - Shared-spool memory consumption greatly reduced (by 70%) - Crashing when syslog > 2G fixed (Linux only) - Empty syslogs no longer emit critical errors -------------------------------------------------------------------------------- 1.0.2 - Custom MessageID search depths per feed object with: MessageIDDepth MessageIDPath path/to/table - Backend connection pool automatically managed, use of NumberOfStreams in backend Feed objects now discouraged - CascadeFeed introduced - Addition of "Megabytes", "Gigabytes" and "Terabytes" for article spool sizes (Terabytes not currently used, it is for later addition of large spools) - Large spool supported - "RetentionGroups" and "RetentionHours" bugs fixed - Tornado and validate_tornado coring on FilterSubscription fixed - Command "article" followed by command "body" without arguments causing Tornado to dump core fixed - Command "next" followed by command "head" failing in Tornado fixed -------------------------------------------------------------------------------- 1.0.1 - Shared spools are now opened read-only - Added "RetentionPriority (1-255)" to feeds, to explicitly tell what priority this feed's retention should be considered. - Re-engineering of the overview system to match the unique nature of tornado's purpose. Addition of OverviewPaths, HeaderPaths and TrackPath allow automatic sizing and striping of headers and overview information This change increases has dramatically increased speed and locatlity (500%) - Addition of "Gigabytes" and "Terabytes" for article spool sizes (Terabytes not currently used, it is for later addition of large spools) - DelayTCPFailover feed directive added - Shared spools are now mounted read-only. Previously they were used read-only, but opened in read/write mode. -------------------------------------------------------------------------------- 1.0.0 - Backfill (populating front end with backend content). - Standardized Naming for all download distribution oraganizations/additions. - Command Line options available in the .conf file. -------------------------------------------------------------------------------- 1.0.0.16 [beta] - Fixed "list active" command. -------------------------------------------------------------------------------- 1.0.0.15 [beta] ***NOTE*** THIS IS A BETA RELEASE of the Tornado Advanced Chaining server, it is for inital performance testing only, and is not certified by Highwinds Software to be suitable for a production environment. -------------------------------------------------------------------------------- Copyright (c) 1999-2008 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com