Compress and Move Log Files

Sometimes log files bog a system down. For one of our servers, I made this little Python script that compresses (via WinRAR) the log files in a directory, and then moves them to a backup location. The only little catch is that I wanted to leave the latest log files for in that directory. Log files are created daily, so the the latest log files have a datestamp of today. Here’s how I did it.

First Create the Python Script:

import os
import datetime
 
dateStamp  = datetime.datetime.now().strftime("%Y-%m-%d") 
imsLogPath = 'd:\\LogFiles\\'                     
# Don't use a mapped drive but use UNC for network drives. Task Schedule seems to choke when it calls Python.
newRARPath = '"\\\\192.168.1.2\\Root\\backups\\' + dateStamp + '.rar"'
rarPath    = '"C:\\Program Files\\WinRAR\\rar.exe" a -m5 ' + newRARPath 
 
# Get Latest Files
smtpLatest   = os.popen(r"dir /od /a-d /b " + imsLogPath + "SMTP*.log").read().splitlines()[-1]
postLatest   = os.popen(r"dir /od /a-d /b " + imsLogPath + "POST*.log").read().splitlines()[-1]
ischedLatest = os.popen(r"dir /od /a-d /b " + imsLogPath + "iSched*.log").read().splitlines()[-1]
relayLatest  = os.popen(r"dir /od /a-d /b " + imsLogPath + "Relay*.log").read().splitlines()[-1]
qengLatest   = os.popen(r"dir /od /a-d /b " + imsLogPath + "Qeng*.log").read().splitlines()[-1]
 
# Get List of All Files
allFiles     = os.popen(r"dir /od /a-d /b " + imsLogPath + "*.log").read().splitlines()
 
# Remove Latest Files from All Files List
allFiles.remove( smtpLatest )
allFiles.remove( postLatest )
allFiles.remove( ischedLatest )
allFiles.remove( relayLatest )
allFiles.remove( qengLatest )
 
# allFiles Array Has the list of files
 
# Flatten Array allFiles to be used as a parameter in system command
flatLogPathList = ""
for filenameWithPath in allFiles:
  flatLogPathList = flatLogPathList + imsLogPath + filenameWithPath + " "
 
 
# Execute WinRar
path = rarPath + " " + flatLogPathList.rstrip()
os.system( '"' + path + '"' )
 
# Delete all log files
os.system( '"del ' + flatLogPathList.rstrip() + '"' )

Then I set up the Scheduled Task:

With these Settings:

Convert Minutes to Hours

I often use both Winamp and my iPhone to listen to music. These two, unfortunately, show the time differently in the songs. Winamp displays the time in minutes (mm) while the iPhone does it hour/minutes (hh:mm). Here’s a quick little script I whipped together because I’m too lazy to do this in my head, especially for audio books where an audio book can be over 500 minutes, and I need to convert to iPhone time because I want to continue listening where I had just left off on Winamp.

use POSIX qw(ceil floor); # used for the floor function
 
sub GetToken {
  # @_ = flatten args list from an array
  # @_[0] = first argument
 
  $data      = @_[0];
  $delimiter = @_[1];
  $token     = @_[2] - 1;
 
  @tokens_array = split($delimiter, $data);   
 
  return @tokens_array[$token]; 
}
 
sub chr_conver_min {  
  if (length(@_[0]) == 1) {
    return "0".@_[0];
  }
  else {
    return @_[0];
  }   
}
 
 
sub iphone_time_convert {
 
  # converts winamp time to iphone - winamp stores time only in minutes.  
  # @_[0]   =  winamp_time, e.g. 124:34
  # $hour   = floor($winamp_time/60);
  # $minute = $winamp_time % 60;
 
  $winamp_hour_min = GetToken(@_[0], ":", 1);  
  $winamp_seconds  = GetToken(@_[0], ":", 2);  
 
  return floor($winamp_hour_min/60).":".chr_conver_min( ($winamp_hour_min % 60) ).":".$winamp_seconds;
 
}
 
 
sub winamp_time_convert {  
 
  # converts iphone time to winamp  
  # @_[0] = iphone_time, e.g. 3:43:34    
  $iphone_hour     = GetToken(@_[0], ":", 1);  
  $iphone_min      = GetToken(@_[0], ":", 2);    
  $iphone_seconds  = GetToken(@_[0], ":", 3);
 
  return (($iphone_hour * 60) + $iphone_min).":".$iphone_seconds;
 
}
 
sub show_help {
  print "\nDisplays the conversion of time between winamp and iPhone.\n\n";
  print "   winamptime [-w2i|-i2p] [mm:ss][hh:mm:ss]\n\n";
  print "Example to convert winamp time to iPhone: \n\n";
  print "   winamptime -w2i 212:41\n\n";
  print "Example to convert iPhone time to winamp, seconds being optional: \n\n";
  print "   winamptime -i2w 2:31:41\n";
  print "   winamptime -i2w 2:31\n\n";
}
 
 
# START
 
# Optimize this:
if( $ARGV[0] eq "-w2i" ) 
{
  # winamp to iphone time
  if ( length($ARGV[1]) > 0 ) {
    print "iPhone Time: ".iphone_time_convert( $ARGV[1] )."\n";
  }
}
elsif( $ARGV[0] eq "-i2w" ) 
{
  # iphone to winamp time
  if ( length($ARGV[1]) > 0 ) {
    print "Winamp Time: ".winamp_time_convert( $ARGV[1] )."\n";
  }
}
else 
{
  show_help();
}

Output:

SQL Server 2008 Installation Errors

Recently I’ve had to install SQL Server 2008 on a few machines. Some of these errors made absolutely no sense, but I found solutions that worked for me. Maybe they’ll help you.

Windows 2000 Server (32-bit)


“The procedure entry point GetConsoleProcessList could not be located in the dynamic link library KERNEL32.dll.”

That sounds pretty self-explanatory, right? Actually, this was the easiest error to make sense of. This is the error you get because SQL Server 2008 is not compatible with Windows 2000 Server. It comes from the fact that SQL Server 2008 needs .NET 3.5+, and that version of .NET is not Windows 2000-compatible.

Windows XP and Windows 2003 Server (32-bit)


Surprisingly, I received no errors while trying to install it on Windows XP. Everything went fine with Windows 2003 Server.

Windows 7 (64-bit)


OK, this was hell on earth. First I ran the setup, and mid-way through the installation, I received this gem:

“SQL Server Setup has encountered the following error:

Invoke or BeginInvoke cannot be called on a control until the window handle has been created..”

I noticed that this arises if you have the focus on another Window while the installer is running. I was browsing the web via Firefox and I received this. To make sure you don’t get this, don’t have any other Window running and only have the focus on the installer.

That wasn’t the only error I received. When I initially ran the setup, I saw a console window flash and then nothing would happen. (I waited for a few minutes and nothing. Setup.exe wasn’t even in the task manager.) When you run setup.exe, it fires landingpage.exe, which is the actual wizard installer. Unfortunately, it wasn’t running it. I even made sure I had the requirements, and nothing. I rebooted several times, and finally decided to run landingpage.exe directly. When I ran it, it finally saw the wizard. I was joyful, until it got to this part:

What the heck?! Where are the features?

Why would the wizard make it this far and then then not have the features to let me install? This was useless. If you’re curious, it should’ve looked like this:

Then for the heck of it, I decided to Install SQL Server 2008 SP1, without even installing SQL Server 2008 first. Don’t know why, but it let me install it anyways. Then when I ran the SQL Server 2008 installer again, but this time, I would see these features:

Ugh, still no luck. Finally, the only thing that I could conclude was that I may have a bad installation. So I restored my machine to an earlier restore point (to undo this installation mess) and re-downloaded the installation files.

Voila! No more problems. So that’s what it was. I was finally able to see all the features to install. So maybe the files got corrupted upon download or it was an incomplete download.

Java Pairs Well with Which Database?

In the same way there’s a tight bond between MySQL and PHP, SQL Server and ASP.NET, SQL Server and ColdFusion – what goes well with Java? Oracle? Being curious, I started searching in employment web sites. I searched for “Java” and one of these databases: Oracle, MySQL, SQL Server and PostgreSQL. (I put in “SQL Server” using quotes.) The sites used were: craigslist, Monster.com, Dice.com, and Yahoo! Hotjobs.

The numbers signify how many job entries were returned.

So it does seem Oracle goes with Java. Also I noticed how many people call “SQL Server” just “SQL.” Sort of confusing and hard to tell if they’re referring to the platform or language.

SQLCompare via Command Console

If you haven’t heard of SQLCompare by RedGate, you’re missing out. It’s an amazing product. In summary, you can do the following (which I use it for 99% of the time):

  • Compare Schema / TSQL code from Different Databases
  • Sync Schema / TSQL code across two databases
  • Generate TSQL code from your comparison so you can use it for deployments
  • Generate reports on SQL Changes

Most of those things can be done via the GUI version of the tool. The product also comes with a command line version, which you can use it in your build script.

To script out your entire database via the command console, you can do the following:

SQLCompare /force /database1:YOURDATABASENAME /username1:sa /password1:password /server1:YOURDATABASESERVERNAME /makescripts:c:\x

To compare two databases and generate a report via the command console, you can do the following. The reason there’s so many switches is because you need to enter the database name and credentials for the two databases.

SQLCompare /force /database1:DB1NAME /username1:sa /password1:password /server1:SERVER1NAME /database2:DB2NAME /username2:sa /password2:password /server2:SERVER2NAME /report:c:\report.html /reporttype:Interactive

Logical vs Physical ER Diagrams

Logical diagrams are to convey requirements only. Physical diagrams represent the actual data structure to support the requirements and take into account technical scalability and speed.

Edit: I hate the way I had to format this document for this blog post. If you want this tutorial better formatted, check out the Word document.

One-to-Many Relationship


Logical

On ER/Studio, two tables are created. The Store table has a primary key StoreID. The Manager table has a primary key, ManagerID and a foreign key, StoredID (which is mapped to StoreID from the Store table).

Physical

On SQL Server, two tables are created. The Store table has a primary key StoreID. The Manager table has a primary key, ManagerID and a foreign key, StoredID (which is mapped to StoreID from the Store table).

Manager Table:

If you allow NULLs for StoreID in the Manager table, then you’ll be able to have a Manager without a store. If you don’t allow NULLs (leave it unchecked), then you’ll have to have at least one Store assigned to a Manager.

Querying

The above states that one store can have many managers. Here’s some sample data what’s in the tables:

SELECT * FROM Manager

SELECT * FROM Store

Get all manager information with for all managers that belong to a store:

SELECT  Manager.ManagerID,  Manager.FirstName,  Manager.LastName,  Manager.StoreID,
        Store.[Name],  Store.Address,  Store.State,  Store.City,  Store.Zip
FROM    Manager
        INNER JOIN Store ON Manager.StoreID = Store.StoreID

Get all manager information with for all managers (even if they don’t have a store):

SELECT  Manager.ManagerID,  Manager.FirstName,  Manager.LastName,  Manager.StoreID,
        Store.[Name],  Store.Address,  Store.State,  Store.City,  Store.Zip
FROM    Manager
        LEFT OUTER JOIN Store ON Manager.StoreID = Store.StoreID

Notice the NULL for Steamboat Willie. He doesn’t have a store, so all Store related fields show as NULL.


Many-to-Many Relationship


In order to implement this physically, you need a join table. In this case, we use StoreManager. Logically, you only need only two entities (Store and Manager).

Logical

Physical


On SQL Server, three tables are created. The Store table has a primary key StoreID. The Manager table has a primary key, ManagerID. The table StoreManager has two foreign keys:  StoredID (which is mapped to StoreID from the Store table) and ManagerID (which is mapped to the ManagerID from the Manager table).

StoreManager Table:

If you allow NULLs for StoreID and ManagerID in the StoreManager table, then you’ll be able to have a Manager without a store. If you don’t allow NULLs (leave it unchecked for both), then you’ll have to have at least one Store assigned to a Manager.

Here’s some sample data what’s in the tables:

SELECT * FROM Manager


SELECT * FROM Store


SELECT * FROM StoreManager


Get all manager information associated with his store:

SELECT  Manager.ManagerID,  Manager.FirstName,  Manager.LastName,
        StoreManager.StoreID, StoreManager.ManagerID, Store.StoreID,
        Store.[Name],Store.Address, Store.State, Store.City, Store.Zip
FROM    Store
        INNER JOIN StoreManager ON Store.StoreID = StoreManager.StoreID
        INNER JOIN Manager ON StoreManager.ManagerID = Manager.ManagerID

Custom Fonts with CSS3

One great thing about CSS3 is its typography facilities. With that comes the ability to use system fonts other than the safe-web fonts such as Arial/Helvetica, Times New Roman/Times, or Courier New/Courier. Actually, it’s been a feature since CSS2, but at that time, but not all browsers have been supporting the same type of font file. ( Man, why can’t browsers agree on *SOMETHING*.) You can read more about the font files that are supported (EOT, OTF, TTF), but for the most part, as always IE is the buzz-kill. Currently, most recent-versioned browsers support the TTF format except IE8.

Anywhoot, here’s a demo that loads a TTF Upside-down font and applies the font to a text area. It lets you type upside-down. Code is very simple:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Custom Text</title>
<style type="text/css">
@font-face 
{
	font-family: "upsidedown_font";	
	src: url("http://www.shinylight.com/wp-content/uploads/2010/01/Stagdive.ttf");
}
 
textarea 
{
	font-family: "upsidedown_font", Arial; 
	font-size:48px;
}
</style>
 
</head>
 
<body>
<h3>Type stuff. It'll be upside down.</h3>
<p>Only supported in modern-versioned browsers. Not supported in IE8. </p>
<textarea cols="60" rows="10"></textarea>
</body>

World’s Smallest Program

Ouch, I guess the dream I had for having the world’s smallest program has been crushed. I made a program a long long time ago in Assembly using Windows 95’s DEBUG.EXE that rebooted the computer. The reason why it’s so small is that it reboots the computer, so once the BIOS Interrupt executes, there’s no need to do any other clean-up or maintenance. Unfortunately after Windows 95, Microsoft got smart and disabled DOS from executing these interrupts from DEBUG.EXE. Here was the code:

N REBOOT!.com
A 100
INT 19
 
R CX
2
W
Q

But what I really want is to see the code for this 0-byte code program written in C.

Recursion vs For-Loop

So I’m currently in process of reading the infamous “Code Complete” by Steve McConnell. So far it’s been an amazing book and I definitely guarantee it to any programmer out there. I’ve just read the section on recursion and it mentioned how doing recursion for a factorial (or fibonacci) function is not as efficient as a for-loop iteration. I guess I never thought about it, since in computer science I was always shoved recursion down my throat when doing factorials. I agree with him that computer science professors are eager to apply the idea of recursion on factorials, but I’ve never remembered a professor mention that it’s not the most efficient way. McConnell states in the book that doing recursion in factorials:

  1. Is not as fast as a for-loop.
  2. Not as clear to read as a for-loop.
  3. Use of run-time memory is unpredictable.

Just for fun, I wanted to test his point on speed. This is a Python script that tests the average speed of a factorial using a for-loop or recursion. I noticed that for numbers less than 3000! the time it took for both functions were exactly the same. It was only when I bumped it up to 5000!, which is a huge number (16,327 digits). Luckily Python lets you work with very large numbers easily. Just had to increase the number of recursion calls in Python from the default 1000.

 
 
import win32api
import sys
 
sys.setrecursionlimit(10000)
 
def factorial_forloop( n ):
  count = 1
  for i in range( n, 0, -1 ):
    count = count * i  
  return count
 
 
def factorial_recursion(n):
  if n == 0:
     return 1
  else:
     return n * factorial_recursion(n-1)
 
 
 
total_time_recursion = 0
total_time_forloop   = 0
number_of_tries      = 500
 
for i in range( 1, number_of_tries ):
 
  start = win32api.GetTickCount()
  factorial_recursion( 5000 )
  end = win32api.GetTickCount()
  total = end - start  
  total_time_recursion += total
 
 
  start = win32api.GetTickCount()
  factorial_forloop( 5000 )
  end = win32api.GetTickCount()
  total = end - start
  total_time_forloop += total  
 
 
print "\n"  
print "Average time for recursion: ", ( total_time_recursion / 10 ) * .001
print "Average time for for-loop: ", ( total_time_forloop / 10 ) * .001

So in 500 tries, the results were as follows:

Average time for recursion:  1.284 seconds
Average time for for-loop:   1.083 seconds

It doesn’t seem by much but the results are interesting. But then again, a factorial is a very simple algorithm. In future posts I’ll try to test more complicated algorithms and see how they battle out. Also, this is Python. The results for C, C++, or Java may differ.

Generate DateTime based on Time Offset

So let’s say you have a table with a column of type DateTime. Now you have to support timezones. The first thing you create is a table of time offsets (Google Time Offsets). The table would look like this:

Here’s schema for it. You can download the full script with the data as well.

CREATE TABLE [dbo].[TimeZones](
	[TimeZoneID] [int] IDENTITY(1,1) NOT NULL,
	[Offset] [varchar](15) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
	[TimeLabel] [varchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
 CONSTRAINT [PK_TimeZones] PRIMARY KEY CLUSTERED 
(
	[TimeZoneID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO

Now for the purpose of this tutorial, let’s create an event table. This will have a set of dates/times which we’ll use to add the timeoffsets to. The table looks like this:

and here’s the script:

CREATE TABLE [dbo].[TimeEvent](
	[TimeEventID] [int] IDENTITY(1,1) NOT NULL,
	[EventName] [varchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
	[EventDateTime] [datetime] NULL,
	[TimeZoneID] [int] NULL,
 CONSTRAINT [PK_TimeEvent] PRIMARY KEY CLUSTERED 
(
	[TimeEventID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]

Now let’s INNER JOIN the tables properly adding the timeoffset to the event datetime:

SELECT  tz.TimeZoneID,
        tz.Offset,        
        te.TimeEventID,
        te.EventName,
        te.EventDateTime AS 'OriginalEventDateTime',
 
        -- Hour - Extract Hour from Offset
        SUBSTRING(Offset, 0, CHARINDEX(':', Offset, 0)) AS 'Hour',
 
        -- Minute - Extract Minute from Offset        
        SUBSTRING(Offset, CHARINDEX(':', Offset, 0) + 1, LEN(Offset)) AS 'Minute',
 
        -- New Date with UTC Addition
        DATEADD(HOUR,
                CAST(SUBSTRING(Offset, 0, CHARINDEX(':', Offset, 0)) AS INT),
                DATEADD(MINUTE,
                        CAST(SUBSTRING(Offset, CHARINDEX(':', Offset, 0) + 1,
                                       LEN(Offset)) AS INT), EventDateTime -- DateTime change via Offset
                         )) AS 'EventDateTimeWithTimeZone'
FROM    TimeZones tz
        INNER JOIN TimeEvent te ON tz.TimeZoneID = te.TimeZoneID

Here’s the result: