Wednesday, October 30, 2013

cloud renting

Google App Engine https://developers.google.com/appengine/docs/whatisgoogleappengine

http://www.rackspace.com/cloud/servers/
http://4sysops.com/archives/amazon-ec2-pricing-for-dummies-part-1-only-pay-for-what-you-use/
"Basically, there are four major factors to remember when it comes to EC2 price calculation: S3 storage, EBS storage, EC2 bandwidth, and EC2 instance type. In my case, the storage costs are negligible ($2), the bandwidth costs are significant ($15) and the EC2 instance costs are decisive ($82). "

cloud rate calculator
  1. In general, one database = one instance. You spin up instances, and do what you like with them. Definitely possible to have more connections to it.
  2. Hours per month is just that. How many hours per month you have the instance active. If you plan to have the instance active 24/7, you may find more cost effective alternatives with other cloud providers. If you run it less often than that, you save money when it's not active. It's billed hourly to your account at the rate specified.
  3. Upload data is counted at the standard transfer rates. A few GBs doesn't cost much, but you will be paying for the service starting the moment you spin up the instance.
"simple" monthly calculator http://calculator.s3.amazonaws.com/calc5.html
Database Service Pricing http://aws.amazon.com/rds/pricing/
http://bitnami.com/cloud-cost-calculator
There are a lot of nice UI calculators (newvem, trusted advisor, ylastic, etc) that will do it for you. Personally I developed one for myself (it is free for everybody) and use it to analyze AWS accounts of my clients to generate a cost report for various AWS services (EC2, RDS, EBS, ELB, S3, etc...).

http://stackoverflow.com/questions/158765/amazon-ec2-cost
Rackspace delivers a better service that Amazon AWS at a fraction of the cost. A basic Rackspace Cloud Server (dedicated only to you) costs around $11/mo and their customer service is astoundingly good. (For example, you can actually TALK to someone via phone or live chat, instead of having to post in community support forums. With amazon you have to subscribe to an annual service contract in order to talk to anyone, which costs around $250/year)
As of 2012, this answer is mostly incorrect. Both Rackspace and Amazon EC2 give you a virtual machine; so on both services you are sharing the physical hardware. For small deployments, Amazon is much cheaper than Rackspace because Amazon is FREE for the first year using micro instances. Even after that, for hosting a web server Amazon is still cheaper when you avail yourself of reserved instances/spot instances. Rackspace does have a more compelling support offering,
Although AWS seems to be inexpensive at the begining, but in reality it is very expensive.
After using AWS for some time, I discovered it is actually very expensive, you better rent one or more dedicated servers. which will give you more processing power and will cost much less.AWS costs will shoot in the sky if your website gets many visitors, serve many files and requests... these small tiny fractions that they charge sum up. Examples: last month I used AWS, I paid 10$ just for requests into S3, which means they count how many requests you made and sum them up and charge you for this... but you still have to pay for the actual bandwidth of these requests! which sounds crazy for me.. why do I have to pay for the requests if I already pay for bandwidth used & for storage costs?you also have to pay for IO requests! which means everytime you read from or write to the hard drive you they are adding up small fractions!these small fractions may look very small, but if your website is very busy, you will be surprised at the end of the monthCosts: instance + traffic /from/to instance + hard drive + requests to hard drive + S3 storage + S3 bandwidth + S3 requeststhe best thing to do is to use S3 as a storage, and server your files from your own servers.. this way you have to pay for storage and your requests + bandwith usage will be minimal....
If you are running a server 24x7 then buy reserved instances. For example a reserved instance of Linux small instance running 24x7 for 3 years will cost a total of $572.36. $257 for purchased of reserved instance then $8.64 per month until the 3 years runs out. An on-demand instance would cost you $43 per month, so reserved is more cost effective for 24x7 machines. (remember, this is a machine you have full control - run many websites - over like a personal box)
If you are doing dev tests, then use a Spot Instance or on-demand. An amazon Linux spot instance is 0.007 per hour right now. An on-demand instance is 0.06 per hour. A spot instance dies when it is turned off, an on-demand can be stopped and started. Spot instances are used by people who need to do short term work like web crawling or census data junk that lasts for a short time.
It costs almost nothing to play around with AWS so you get an idea of the costs. It costs me less than a dollar to play around for 3 days during work hours. I created on-demand machines, I created spot instances and ran them all night, I spent about 12 hours total. Its cheap to learn. You will learn more experimenting than you will asking questions in forums.

Friday, October 4, 2013

change the clock source in the system

  • Red Hat Enterprise Linux 4:
    • Checking the current clock source:
      • One can easily perform this task through searching in /var/log/dmesg file for "time":
        [root@localhost ~]# grep "time" /var/log/dmesg
        ...
        Using HPET for base-timer
        Using HPET for gettimeofday
        Using hpet for high-res timesource
        ...
        
    • Verifying documentation on time source kernel parameters:
      • In order to check what time source may be used with RHEL4 kernel, one should read the kernel documentation file /usr/share/doc/kernel-doc-2.6.9/Documentation/kernel-parameters.txt:
                clock=          [BUGS=IA-32, HW] gettimeofday timesource override.
                                Forces specified timesource (if available) to be used
                                when calculating gettimeofday(). If specified timesource
                                is not available, it defaults to PIT.
                                Format: { pit | tsc | cyclone | pmtmr }
        
                hpet=           [IA-32,HPET] option to disable HPET and use PIT.
                                Format: disable
        
                notsc           [BUGS=IA-32] Disable Time Stamp Counter
        
    • Overriding the clock source configuration:
      • adding clock= to the kernel stanza:
        title Red Hat Enterprise Linux AS (2.6.9-89.ELsmp)
                root (hd0,0)
                kernel /vmlinuz-2.6.9-89.ELsmp ro root=LABEL=/ rhgb quiet clock=pmtmr
                initrd /initrd-2.6.9-89.ELsmp.img
        
  • Red Hat Enterprise Linux 5:
    • Checking the current clock source:
      • Unfortunately, as the original snapshot of RHEL5 kernel was done at a time in which the actual linux-2.6 clocksource code was just being introduced for i386 and x86_64, it is not accurate to list the content of files into /sys/devices/system/clocksource/* directories in order to determine the current and the available system clock sources. Instead, to determine the clock source accurately, one should proceed the same as in RHEL4 way: searching in /var/log/dmesg file:
        [root@localhost ~]# grep "time" /var/log/dmesg
        ...
        time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.
        time.c: Detected 3200.238 MHz processor.
        ...
        
    • Verifying clock source available kernel parameters:
      • In order to check what time source may be used with RHEL5 kernel, one should read the kernel documentation file /usr/share/doc/kernel-doc-2.6.18/Documentation/kernel-parameters.txt:
                clock=          [BUGS=IA-32, HW] gettimeofday clocksource override.
                                [Deprecated]
                                Forces specified clocksource (if available) to be used
                                when calculating gettimeofday(). If specified
                                clocksource is not available, it defaults to PIT.
                                Format: { pit | tsc | cyclone | acpi_pm | tsccount | notsccount }
        
                clocksource=    [GENERIC_TIME] Override the default clocksource
                                Override the default clocksource and use the clocksource
                                with the name specified.
        
                hpet=           [IA-32,HPET] option to disable HPET and use PIT.
                                Format: disable
        
                notsc           [BUGS=IA-32] Disable Time Stamp Counter
        
    • Overriding the clock source configuration:
      • adding clocksource= to the kernel stanza:
        title Red Hat Enterprise Linux (2.6.18-128.el5)
                root (hd0,0)
                kernel /vmlinuz-2.6.18-128.el5 ro root=LABEL=/ rhgb quiet clocksource=acpi_pm
                initrd /initrd-2.6.18-128.el5.img
        
  • Red Hat Enterprise Linux 6:
    • Checking the current clock source:
      • listing the /sys/devices/system/clocksource/*/current_clocksource file content:
        [root@localhost ~]# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
        hpet
        
    • Verifying available clock sources:
      • listing the /sys/devices/system/clocksource/*/available_clocksource file content:
        [root@localhost ~]# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
        hpet acpi_pm
        
      • browsing the documentation file /usr/share/doc/kernel-doc-2.6.32/Documentation/kernel-parameters.txt:
                clocksource=    [GENERIC_TIME] Override the default clocksource
                                Format:                         Override the default clocksource and use the clocksource
                                with the name specified.
                                Some clocksource names to choose from, depending on
                                the platform:
                                [all] jiffies (this is the base, fallback clocksource)
                                [ACPI] acpi_pm
                                [ARM] imx_timer1,OSTS,netx_timer,mpu_timer2,
                                        pxa_timer,timer3,32k_counter,timer0_1
                                [AVR32] avr32
                                [X86-32] pit,hpet,tsc,vmi-timer;
                                        scx200_hrt on Geode; cyclone on IBM x440
                                [MIPS] MIPS
                                [PARISC] cr16
                                [S390] tod
                                [SH] SuperH
                                [SPARC64] tick
                                [X86-64] hpet,tsc
        
                hpet=           [X86-32,HPET] option to control HPET usage
                                Format: { enable (default) | disable | force |
                                        verbose }
                                disable: disable HPET and use PIT instead
                                force: allow force enabled of undocumented chips (ICH4,
                                        VIA, nVidia)
                                verbose: show contents of HPET registers during setup
        
                notsc           [BUGS=X86-32] Disable Time Stamp Counter
        
    • Overriding the clock source configuration:
      • Dynamically and temporarily (not persistent to reboots):
        [root@localhost ~]# echo "acpi_pm" > /sys/devices/system/clocksource/clocksource0/current_clocksource
        
      • Permanently, adding clocksource= to the kernel stanza:
        title Red Hat Enterprise Linux Server (2.6.32-71.18.2.el6.x86_64)
                root (hd0,0)
                kernel /vmlinuz-2.6.32-71.18.2.el6.x86_64 ro root=LABEL=/ crashkernel=auto clocksource=acpi_pm
                initrd /initramfs-2.6.32-71.18.2.el6.x86_64.img
        
  • An overview on hardware clock and system timer circuits:
    When it comes to talk about a system's clock, the hardware sits at the very bottom. Every typical system has several devices, usually implemented by clock chips, that provide timing features and can serve as clocks. So, which hardware is available depends on the particular architecture. The clock circuits are used both to keep track of the current time of the day and to make precise time measurements. The timer circuits are programmed by the kernel,so they issue interrupts at a fixed, and predefined, frequency. For instance, IA-32 and AMD64 systems have at least one programmable interrupt timer (PIT) as a classical timer circuit, which is usually implemented by an 8254 CMOS chip. Let's briefly describe the clock and timer circuits that are usually found with any nearly modern system of those architectures:
    • Real Time Clock (RTC): The RTC is independent of the system's CPU and any other chips. As it is energized by a small battery, it continues to tick even when the system is switched off. The RTC is capable of issuing interrupts at frequencies ranging between 2Hz and 8,192Hz. Linux uses the RTC only to derive the time and date at boot time.
    • Programmable Interrupt Timer (PIT): The PIT is a time-measuring device that can be compared to the alarm clock of a microwave oven: it makes the user aware that the cooking time interval has elapsed. Instead of ringing a bell, the PIT issues a special interrupt called timer interrupt, which notifies the kernel that one more time interval has elapsed. As the time goes by, the PIT goes on issuing interrupts forever at some fixed (architecture-specific) frequency established by the kernel.
    • Time Stamp Counter (TSC): All 80x86 microprocessors include a CLK input pin, which receives the clock signal of an external oscillator. Starting with the Pentium, 80x86 microprocessors sport a counter that is increased at each clock signal, and is accessible through the TSC register which can be read by means of the rdtsc assembly instruction. When using this register the kernel has to take into consideration the frequency of the clock signal: if, for instance, the clock ticks at 1GHz, the TSC is increased once every nanosecond. Linux may take advantage of this register to get much more accurate time measurements.
    • CPU Local Timer: The Local APIC (Advanced Programmable Interrupt Controller) present in recent 80x86 microprocessors provide yet another time measuring device, and it is a device, similar to the PIT, which can issue one-shot or periodic interrupts. There are, however, a few differences:
      • The APIC's timer counter is 32bit long, while the PIT's timer counter is 16bits long;
      • The local APIC timer sends interrupts only to its processor, while the PIT raises a global interrupt, which may be handled by any CPU in the system;
      • The APIC's timer is based on the bus clock signal, and it can be programmed in such way to decrease the timer counter every 1, 2, 4, 8, 16, 32, 64, or 128 bus clock signals. Conversely, the PIT, which makes use of its own clock signals, can be programmed in a more flexible way.
    • High Precision Event Timer (HPET): The HPET is a timer chip that in some future time is expected to completely replace the PIT. It provides a number of hardware timers that can be exploited by the kernel. Basically the chip includes up to eight 32bit or 64bit independent counters. Each counter is driven by its own clock signal, whose frequency must be at least 10MHz; therefore the counter is increased at least once in 100 nanoseconds. Any counter is associated with at most 32 timers, each of which composed by a comparator and a match register. The HPET registers allow the kernel to read and write the values of the counters and of the match registers, to program one-shot interrupts, and to enable or disable periodic interrupts on the timers that support them.
    • ACPI Power Management Timer (ACPI PMT): The ACPI PMT is another clock device included in almost all ACPI-based motherboards. Its clock signal has a fixed frequency of roughly 3.58MHz. The device is a simple counter increased at each clock tick. However the ACPI PMT is preferable to the TSC if the operating system or the BIOS may dynamically lower the CPU's frequency or voltage. When this happens, TSC's frequency changes causing time warps and others side-effects, while the frequency of ACPI PMT does not.

  • Xen Problem

    XenServer is headache http://sysconfig.org.uk/2011/01/so-long-xenserver/

    - clocksource problem https://wiki.debian.org/Xen/Clocksource

    When running no-citrix kernel, sometime running to clocksource problem.
    The clocksource problem appears after you suspend a Linux host and the kernel in the virtual machine starts spewing this:
    Mar  5 09:24:17 co kernel: [461562.007153] clocksource/0: Time went backwards: ret=f03d318c7db9 delta=-200458290723043 shadow=f03d1d566f4a offset=14
    It turns out that the Citrix kernel uses jiffies as the clocksource per default, while Debian uses the xen clocksource.
    So by just running this on the domU fixes the problem:
    echo "jiffies"> /sys/devices/system/clocksource/clocksource0/current_clocksource
    
    There is no need to decouple the clock from the host. To make this change permanent, you need to addclocksource=jiffies to the bootparameters of your domU kernel.

    For fully virtualized (HVM) instances
    To find your current clocksource
    cat /sys/devices/system/clocksource/clocksource0/current_clocksource
    To see available clocksources
    cat /sys/devices/system/clocksource/clocksource0/available_clocksource

    For para-virtualized (PV) instances,
    echo 1 > /proc/sys/xen/independent_wallclock
    In order to ensure the setting sticks on boot, you should also issue a:
    echo "xen.independent_wallclock = 1" >> /etc/sysctl.conf