UnCommentToLineMatching doesn’t work in cfengine-2.2.8
And — once again — cfengine has wasted part of my life. I’ve never had more trouble with software than I’ve had with cfengine. This time I had to delve into the source code to find the problem, and what I found hilights poor coding and even poorer testing.
Stop me if you’ve heard this one already…
For the life of my I can’t get the following editfiles snippet:
{ /etc/syslog-ng/syslog-ng.conf
LocateLineMatching "### Begin Section loghost ###"
UnCommentToLineMatching "### End .*"
}
To do what I expect on the following bit of syslog-ng.conf:
### Begin Section loghost ###
# source s_remote {
# tcp(ip(0.0.0.0) port(514));
# udp(ip(0.0.0.0) port(514));
# };
### End Section loghost ###
### Begin Section loghost_sys ###
# destination d_messages_byhost {
# file("/var/log/$HOST/messages");
# };
#
# log { source(s_remote); destination(d_messages_byhost); };
### End Section loghost_web ###
I want it to only uncomment the first section. What it does is uncomment the entire rest of the file.
Here’s some interesting debugging output:
Edit action: UnCommentToLineMatching
CommentToRegExp(list,# ### End .*)
Uncomment line # source s_remote {
Uncomment line # tcp(ip(0.0.0.0) port(514));
Uncomment line # udp(ip(0.0.0.0) port(514));
Uncomment line # };
Terminating line: ### End Section loghost ### (Done)
Uncomment line # destination d_messages_byhost {
Uncomment line # file("/var/log/$HOST/messages");
Uncomment line # };
Uncomment line #
Uncomment line # log { source(s_remote); destination(d_messages_byhost); };
It looks like it detects the correct line on which to Terminate, but then Uncomments the rest of the file anyway.
From UnCommentToRegExp() in item-ext.c (line 1282 in cfengine-2.2.8):
for (ip = CURRENTLINEPTR; ip != NULL; ip = CURRENTLINEPTR)
{
if (ip == ip_end)
{
EditVerbose("Terminating line: %s (Done)\n",ip->name);
done = true;
}
“done” is a local variable that is not used anywhere else in UnCommentToRegExp(). Based on the incorrect function name in the Debug2() statement, I take it UnCommentToRegExp() was copied from CommentToRegExp() and the if (done) { break; } was lost somewhere along the way. I attempted to see if this is also the case in cfengine-3.0.0a3, but UnCommentToRegExp() has been removed entirely, it seems.
This also raises a larger question — will cfengine-2.2 be bugfixed after 3.0.0 is released, or will I have to upgrade to 3.0.0 to fix this issue? I certainly don’t want to upgrade to 3.0.0. I’m there’ll be no limit to the number of bugs introduced in that major release.
postfix nags about a domain in BOTH relay_domains and virtual_alias_domains, when it isn’t
I’ve been struggling with the following warning in postfix for a while now:
postfix/trivial-rewrite[xxxx]: warning: do not list domain subdomain.blarg.org in BOTH virtual_alias_domains and relay_domains
The problem is, I’m not listing it in both. I swear.
% postconf relay_domains relay_domains = $mydestination % postconf mydestination mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain % postconf virtual_alias_domains virtual_alias_domains = subdomain.blarg.org
I finally tracked it down today to a relatively obscure (to me, anyway) feature in postfix that causes subdomains to be implicitly matched for certain coniguration parameters. From postconf(5):
parent_domain_matches_subdomains (default: see postconf -d output)
What Postfix features match subdomains of “domain.tld” automatically, instead of requiring an explicit “.domain.tld” pattern. This is planned backwards compatibility: eventually, all Postfix features are expected to require explicit “.domain.tld” style patterns when you really want to match subdomains.
This was exactly what was happening in my case. I was being explicit in my relay_domains and mydestinations, but postfix was matching subdomains for relay_domains:
% postconf parent_domain_matches_subdomains parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks,permit_mx_backup_networks,qmqpd_authorized_clients,relay_domains,smtpd_access_maps
Since this is a feature with planned obsolescence that I’m not really using anyway (afaik), I just completely disabled it:
# Fixes the "do not list domain in BOTH" nags, also the future default behavior # To enable subdomain matching, use .domain.com parent_domain_matches_subdomains =
Apparently it’s still possible to explicitly use subdomain matching by specifying the domain as .blarg.org, which makes things a lot more self-evident, anyway.
Server load balancing in pfSense 1.3
I was poking around at the 1.3-ALPHA-ALPHA pfSense, and it looks like slbd is going to be replaced with OpenBSD’s relayd! I can’t tell you how happy I am about this development. Not only will relayd increase pfSense’s feature set by some order of magnitude, I have hope that it will make pfSense a viable load balancer.
Though pfSense has load balancing features right now, its limitations make it mostly unusable for my applications. The way that slbd operates breaks session stickiness. I’m not sure why, but I suspect that slbd is updating its pf anchor unconditionally per polling period which causes pf to reset its session state. In addition to breaking stickiness, this causes load to be balanced unfairly.
I’ll have to do some testing to see if relayd has this say behavior. If nothing else, though, I’m excited about a whole new host of features, some of which I haven’t seen since I last touched a NetScaler:
- Layer 7 (in addition to Layer 3) load balancing
- Layer 7 health monitoring (http request/response checking)
- In-Flight HTTP request/response modifications (such as appending Remote-Host header)
- Additional balancing algorithms, including one that can be sticky on HTTP headers/cookies
All-in-All pretty fantastic. I don’t imagine we’ll see 1.3 until Q2 of 2009, though, so it may be worth back-porting it as a package for 1.2.1.
What MAC address should I use for my virtual server?
I’m creating a new virtual server, and I need a MAC address to assign to it. (My provisioning/build system requires pre-allocated MACs.) Instead of pulling one out of thin air, I thought I’d look up what address range I should be using. It seems surprisingly difficult to find a straight answer on this one, though.
Locally Administered Addresses
It looks like the 802 standard includes provisions for locally administered addresses versus universally administered addresses. If the second bit transmitted on the wire is a 1, then the address is considered locally administered. If the second bit is a 0, then address is globally administered and the first 24-bits must be a globally unique OUI assigned by the IEEE.
Note that this is the second bit on the wire. It would be the second least-significant bit of the most significant byte when represented in hex. That is, 02:xx:xx:xx:xx:xx is local while 04:xx:xx:xx:xx:xx is universal.
Great, so half of the entire MAC address space is available for me to choose from, right? I think so, but I haven’t really seen used with virtual servers.
IEEE Private OUI
The IEEE is the authority responsible for keeping the first 24 bits of the address unique. Vendors apply for an Organizationally Unique Identifier (OUI) which becomes the first 24 bits. The IEEE has assigned AC-DE-48 to “PRIVATE”. This looks to be the MAC address equivalent of RFC1918 private IP addresses (10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12).
Which should I use?
As best as I can tell, locally administered addresses were intended for the purpose of overriding a burned-in/firmware-provided assigned address. In my case of assigning to a virtual server, I’m trying to emulate an assigned address, not override one. Using the IEEE private OUI seems like a better fit.
Then I still have the option of overriding my privately unique address with a locally assigned address. Because that seems like a good idea. Or something.
Update: Xen Virtuals
It looks like XenSource has registered 00-16-3E with IEEE for use with Xen Virtuals as well. XenSource recommends using this range. ymmv.
mtr versus pot (plain ole traceroute)
It drives me nutso when someone reinvents the wheel and then the reinvention becomes more popular than the actual wheel. Usually it happens with some kind of great slogan explaining why this wheel is better. “New WHEEL! Now with more ROUND!” or “Much like previous so-called WHEELs, but NEW WHEEL is avaiable in 5 pastel colors!”
With that in mind, I finally broke down and took a look at Matt’s Traceroute, mtr. (Ok, ok. Now it’s “My Traceroute”, but that’s even dumber than “Matt’s Traceroute”.)
According to MTR’s website, “mtr combines the functionality of the ‘traceroute’ and ‘ping’ programs in a single network diagnostic tool”. They go on to explain that mtr sends out ICMP echo requests with modified TTLs in order to discover the routers along a route and determine statistics for each router. Fantastic. So it’s traceroute -I.
In general, I’m not a fan of traceroutes that use ICMP instead of UDP. I’ve found ICMP too often receives special-case low-priority processing that provides misleading results. It used to be that ICMP was filtered more rigorously than UDP, but (strangely) this seems to be diminishing. Also programs generating ICMP have to be run as the superuser or left setuid, and who wants that?
The UI to MTR is pretty compelling, though. I thought I was going to hate the ncurses interface, but it turns out that I only want to hate it. You know.. like I want to hate top but end up using it anyway. The point of MTR seems to be to run a traceroute over and over. That’s pretty neat, though it’s something I hardly ever need. MTR also does a better job of displaying multiple routers per hop as well.
Other than the interface, I don’t see much value-add. There’s a lot less in the way of configurable options. The other side of that coin is that I’d really have to stretch to fabricate reasons not to use it as well. For general purpose use, the only really annoying thing is that you have to run it as root.
I’ll probably continue to avoid it for the same reason I avoid ruby-on-rails. Until I need it, of course.
pfSense’s poorly documented ftp proxy
It turns out that when you (a) disable the “disable ftp-proxy” on the WAN interface and (b) create a NAT rule that uses port 21, the ftp proxy will be automatically (and silently) started in the background. If you then modify that NAT rule, the ftp proxy application will not be updated (presumably until a reboot), but if you delete and create the rule again the ftp proxy will be started anew with the updated information.
Forcing CARP failover using pfSense
Anything that has a master/slave cluster should have some sort of mechanism for failing from the master to the slave, right? We’ll see…
pfSense provides HA via OpenBSD’s (very excellent) CARP. Of course I could just increment the advskew on all of the interfaces to fail them over individually, but that doesn’t really work for me. That involves spending some amount of time in a partially failed-over state. Having quite a bit of experience with OpenBSD, I expected there to be a more elegant solution and I certainly wasn’t disappointed. It looks like OpenBSD provides interface groups and a setting called “carpdemote”. Incrementing carpdemote on the master’s “carp” group would cause all carp interfaces to fail over to the next-in-line. nice.
I guess I was just expecting FreeBSD to have the same functionality. FreeBSD 7.0 does have interface groups, but I can’t find carpdemote in the documentation. Still, maybe pfSense 1.2.1 will be worth the upgrade afterall.
All-in-all, forcing the failover one at a time wasn’t so bad. Upping the advskew on all interfaces to 200 from a shell loop like the following got the job done, but that race condition just doesn’t sit well.
for i in 0 1 2 3; do
ifconfig carp$i advskew 200
done
Then I did my work and rebooted the master. When the master came up, all of the interfaces were reset to an advskew of 0, which is less-than-ideal-but-still-expected. Strangely, this didn’t cause the interfaces to fail back to the master. For about 5 minutes the master interfaces remained BACKUP even though the master interfaces had an advskew of 0 and the backup had an advskew of 100.
After about 5 minutes all of the interfaces failed back to the master. I expect this is just a relic of how the advskew parameter actually works by tuning carp announcement intervals.
Free Webmail Username Restrictions
What criteria determine a valid username for the top web mail providers? This seems like it would be useful for filtering that list of a couple hundred thousand e-mail addresses we’ve all got lying around.
Yahoo! Mail Basic Restrictions (yahoo.com, ymail.com, rocketmail.com):
- 4 to 32 characters
- Starts with a letter
- Letters, numbers, underscores and one period (.)
- No consecutive underscores or period, or combination of the two ([_.]{2}
- Cannot end with underscore or period
Regex for invalid username:
(?:^.{,3}$|^.{33,}$|^[^a-z]|[^a-z0-9_.]|\..*\.|[_.]{2}|[_.]$)
Gmail Basic Restrictions (gmail.com):
- 6 to 30 characters
- Only letters, numbers and periods
- Starts and ends alphanumeric
- No consecutive periods (also required by RFC 2822)
Because of the restriction on the character set (#2), RFC 2822 compliance guarantees #3 & #4. As such, these won’t appear in the following regex.
Regex for invalid username:
(?:^.{,5}$|^.{31,}$|[^a-z0-9.])
Windows Live Restrictions (hotmail.com, live.com):
- 4 to 32 characters
- Only letters, numbers, periods, hyphens and underscores
- Cannot have the character sequence “fuck”
- Starts with alphanumeric
- Cannot end with a period (also required by RFC 2822)
The minimum length was just a guess on my part. It’s not published, but 4 is the smallest free user name I could find. I’m sure there are words besides “fuck” that aren’t allowed, but you get the point
(?:^.{,3}$|^.{31,}$|[^a-z0-9._-]|fuck|^[^a-z0-9])
MobileMe Restrictions (me.com, mac.com):
- 3 to 20 characters
- Only letters, numbers, periods, hyphens and underscores
Note: The regex on this page are intended for case-insensitive, un-anchored searches.
tech stuff. 