SCSI and FibreChannel Hotswap for linux 2.5.44-bk2
Subsystems (2)
PATCH 1/1 SCSI and FibreChannel Hotswap for linux 2.5.44-bk2
Linus,
For the last several weeks, I've been posting FibreChannel and SCSI hotswap driver for the 2.5.44 tree to linux-scsi and linux-kernel. The patch includes complete locking for correct insertion and removal without screwing the SCSI subsystem, fibrechannel support for FC targets (insertion or removal by wwn), scsi support for scsi targets, and complete integration with driverfs.
This patch has been reviewed by Alan Cox, Greg KH, Christoph Hellwig, Patrick Mansfield, Rob Landly, Jeff Garzik, Scott Murray, James Bottomley, Mike Anderson, Randy Dunlap and Patrick Mochel. Some of the developers have suggested they would integrate it into 2.5.45. I've made changes suggested by the above contributors and feel the patch is really as good as its going to get (minus some scsi spin down and cache flushing additions I want to make). These can be made post freeze if I get to them.
Please ignore the patch to the qlogic driver, as it is not integrated into the kernel and is only in my development tree.
To see threads on the subject check out:
Linux 2.5 SCSI and FibreChannel Hotswap Support
A: Steven Dake
M: http://marc.theaimsgroup.com/?t=103558444400002&r=1&w=2
M: http://marc.theaimsgroup.com/?t=103542041200001&r=1&w=2
M: http://marc.theaimsgroup.com/?t=103541590500001&r=1&w=2
M: http://marc.theaimsgroup.com/?t=103462115700001&r=1&w=2
M: http://marc.theaimsgroup.com/?t=103429968400002&r=1&w=2
D: http://sourceforge.net/projects/atca-hotswap
MAINTAINERS +6 -0
@@ -1412,6 +1412,12 @@ W: http://www.kernel.dk S: Maintained +SCSI HOTSWAP DRIVER +P: Steven Dake +M: [email protected] +L: [email protected] +S: Maintained + SCSI SG DRIVER P: Doug Gilbert M: [email protected]
drivers/scsi/Config.help +10 -0
@@ -1415,3 +1415,13 @@ whenever you want). If you want to compile it as a module, say M here and read <file:Documentation/modules.txt>. +CONFIG_SCSIFCHOTSWAP + If you want to support the ability to include the hotswap FibreChannel + and SCSI driver, please say yes here. Hotswap can then be executed + through a devicefs interface in /bus/scsi/hotswap_commands. + + This software supports insertion and removal by WWN for FibreChannel + drivers which support this feature. + + The only FibreChannel driver that supports this feature is Qlogic V6 + with a specific support patch.
drivers/scsi/Config.in +1 -0
@@ -1,5 +1,6 @@ comment 'SCSI support type (disk, tape, CD-ROM)' +dep_tristate ' SCSI hotswap support' CONFIG_SCSIFCHOTSWAP $CONFIG_SCSI dep_tristate ' SCSI disk support' CONFIG_BLK_DEV_SD $CONFIG_SCSI dep_tristate ' SCSI tape support' CONFIG_CHR_DEV_ST $CONFIG_SCSI
drivers/scsi/Makefile +1 -0
@@ -118,6 +118,7 @@ obj-$(CONFIG_ARCH_ACORN) += ../acorn/scsi/ +obj-$(CONFIG_SCSIFCHOTSWAP) += hotswap.o obj-$(CONFIG_CHR_DEV_ST) += st.o obj-$(CONFIG_CHR_DEV_OSST) += osst.o obj-$(CONFIG_BLK_DEV_SD) += sd_mod.o
drivers/scsi/hosts.c +1 -0
@@ -371,6 +371,7 @@ scsi_hosts_registered++;spin_lock_init (&shost->default_lock); +sema_init (&shost->host_queue_sema,1 );scsi_assign_lock (shost, &shost->default_lock);atomic_set (&shost->host_active,0 );
drivers/scsi/hosts.h +7 -0
@@ -261,6 +261,12 @@*/ int can_queue; +/* +* Used to determine the id to send the inquiry command to during +* hot additions of FibreChannel targets +*/ +int (*get_scsi_info_from_wwn)(int mode,unsigned long long wwn,int *host,int *channel,int *lun,int *id); +/* * In many instances, especially where disconnect / reconnect are * supported, our host also has an ID on the SCSI bus. If this is
@@ -364,6 +370,7 @@*/ struct list_head sh_list;Scsi_Device * host_queue; +struct semaphore host_queue_sema;struct list_head all_scsi_hosts;struct list_head my_devices;
drivers/scsi/hotswap.c +921 -0
921 lines not on this page · raw message
drivers/scsi/qla2xxx/qla2x00.c +76 -0
94 lines not on this page · raw message
drivers/scsi/scsi.c +39 -0
@@ -407,6 +407,9 @@* allow us to more easily figure out whether we should * do anything here or not. */ + +down (&host->host_queue_sema); +for (SDpnt = host->host_queue; SDpnt; SDpnt = SDpnt->next) {
@@ -424,6 +427,9 @@break ; } } + +up (&host->host_queue_sema); +if (SDpnt) {/* * Some other device in this cluster is busy.
@@ -1693,6 +1699,8 @@ len += size; pos = begin + len;#endif +down (&HBA_ptr->host_queue_sema); +for (scd = HBA_ptr->host_queue; scd; scd = scd->next) {proc_print_scsidevice (scd, buffer, &size, len); len += size;
@@ -1705,6 +1713,8 @@if (pos > offset + length) goto stop_output; } + +up (&HBA_ptr->host_queue_sema); } stop_output:
@@ -1863,6 +1873,8 @@if (!HBA_ptr) goto out; +down (&HBA_ptr->host_queue_sema); +for (scd = HBA_ptr->host_queue; scd; scd = scd->next) { if ((scd->channel == channel && scd->id == id
@@ -1871,6 +1883,8 @@ } } +up (&HBA_ptr->host_queue_sema); + err = -ENOSYS ;if (scd) goto out;/* We do not yet support unplugging */
@@ -1909,6 +1923,8 @@if (!HBA_ptr) goto out; +down (&HBA_ptr->host_queue_sema); +for (scd = HBA_ptr->host_queue; scd; scd = scd->next) { if ((scd->channel == channel && scd->id == id
@@ -1917,6 +1933,8 @@ } } +up (&HBA_ptr->host_queue_sema); +if (scd ==NULL ) goto out;/* there is no such device attached */
@@ -1950,9 +1968,14 @@if (scd->prev !=NULL ) scd->prev->next = scd->next; +down (&HBA_ptr->host_queue_sema); +if (HBA_ptr->host_queue == scd) { HBA_ptr->host_queue = scd->next; } + +up (&HBA_ptr->host_queue_sema); +blk_cleanup_queue (&scd->request_queue);if (scd->inquiry)kfree (scd->inquiry);
@@ -2001,11 +2024,15 @@for (shpnt =scsi_host_get_next (NULL ); shpnt; shpnt =scsi_host_get_next (shpnt)) { +down (&shpnt->host_queue_sema); +for (SDpnt = shpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {if (tpnt->detect) SDpnt->attached += (*tpnt->detect) (SDpnt); } + +up (&shpnt->host_queue_sema); }/*
@@ -2021,6 +2048,8 @@*/ for (shpnt =scsi_host_get_next (NULL ); shpnt; shpnt =scsi_host_get_next (shpnt)) { +down (&shpnt->host_queue_sema); +for (SDpnt = shpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {scsi_build_commandblocks (SDpnt);
@@ -2040,6 +2069,8 @@else scsi_release_commandblocks (SDpnt); } + +up (&shpnt->host_queue_sema); }MOD_INC_USE_COUNT ;
@@ -2074,6 +2105,8 @@for (shpnt =scsi_host_get_next (NULL ); shpnt; shpnt =scsi_host_get_next (shpnt)) { +spin_lock (&shpnt->host_queue_sema); +for (SDpnt = shpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {if (tpnt->detach)
@@ -2090,6 +2123,8 @@scsi_release_commandblocks (SDpnt); } } + +up (&shpnt->host_queue_sema); }/* * Extract the template from the linked list.
@@ -2160,6 +2195,8 @@for (shpnt =scsi_host_get_next (NULL ); shpnt; shpnt =scsi_host_get_next (shpnt)) {printk (KERN_INFO "h:c:t:l (dev sect nsect cnumsec sg) (ret all flg) (to/cmd to ito) cmd snse result\n" ); +down (&shpnt->host_queue_sema); +for (SDpnt = shpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {for (SCpnt = SDpnt->device_queue; SCpnt; SCpnt = SCpnt->next) {/* (0) h:c:t:l (dev sect nsect cnumsec sg) (ret all flg) (to/cmd to ito) cmd snse result %d %x */
@@ -2191,6 +2228,8 @@ SCpnt->result); } } + +up (&shpnt->host_queue_sema); }#endif /* CONFIG_SCSI_LOGGING */ /* } */ }
drivers/scsi/scsi_error.c +21 -0
@@ -202,6 +202,8 @@int devices_failed =0 ; +down (&shost->host_queue_sema); +for (sdev = shost->host_queue; sdev; sdev = sdev->next) {for (scmd = sc_list; scmd; scmd = scmd->bh_next) {if (scmd->device == sdev) {
@@ -227,6 +229,8 @@ } } +up (&shost->host_queue_sema); +SCSI_LOG_ERROR_RECOVERY (2 ,printk ("Total of %d commands on %d" " devices require eh work\n" , total_failures, devices_failed));
@@ -247,6 +251,8 @@Scsi_Device *sdev;Scsi_Cmnd *scmd; +down (&shost->host_queue_sema); +for (found =0 , sdev = shost->host_queue; sdev; sdev = sdev->next) {for (scmd = sdev->device_queue; scmd; scmd = scmd->next) {if (scsi_eh_eflags_chk (scmd,SCSI_EH_CMD_ERR )) {
@@ -283,6 +289,8 @@ } } +up (&shost->host_queue_sema); +SCSI_LOG_ERROR_RECOVERY (1 ,scsi_eh_prt_fail_stats (*sc_list, shost));if (shost->host_failed != found)
@@ -962,6 +970,8 @@SCSI_LOG_ERROR_RECOVERY (3 ,printk ("%s: Trying BDR\n" , __FUNCTION__)); +down (&shost->host_queue_sema); +for (sdev = shost->host_queue; sdev; sdev = sdev->next) {for (scmd = sc_todo; scmd; scmd = scmd->bh_next) if ((scmd->device == sdev) &&
@@ -985,6 +995,7 @@scsi_eh_finish_cmd (scmd, shost); } } +up (&shost->host_queue_sema);return shost->host_failed; }
@@ -1016,11 +1027,15 @@/* * Mark all affected devices to expect a unit attention. */ +down (&scmd->host->host_queue_sema); +for (sdev = scmd->host->host_queue; sdev; sdev = sdev->next)if (scmd->channel == sdev->channel) { sdev->was_reset =1 ; sdev->expecting_cc_ua =1 ; } + +up (&scmd->host->host_queue_sema); }return rtn; }
@@ -1052,11 +1067,13 @@/* * Mark all affected devices to expect a unit attention. */ +down (&scmd->host->host_queue_sema);for (sdev = scmd->host->host_queue; sdev; sdev = sdev->next)if (scmd->channel == sdev->channel) { sdev->was_reset =1 ; sdev->expecting_cc_ua =1 ; } +up (&scmd->host->host_queue_sema); }return rtn; }
@@ -1478,7 +1495,9 @@* now that error recovery is done, we will need to ensure that these * requests are started. */ +down (&shost->host_queue_sema);spin_lock_irqsave (shost->host_lock, flags); +for (sdev = shost->host_queue; sdev; sdev = sdev->next) {request_queue_t *q = &sdev->request_queue;
@@ -1491,7 +1510,9 @@ q->request_fn (q); } +spin_unlock_irqrestore (shost->host_lock, flags); +up (&shost->host_queue_sema); }/**
drivers/scsi/scsi_lib.c +16 -0
@@ -261,6 +261,8 @@if (SDpnt->single_lun &&blk_queue_empty (q) && SDpnt->device_busy ==0 ) {request_queue_t *q; +down (&SHpnt->host_queue_sema); +for (SDpnt = SHpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {if (((SHpnt->can_queue >0 ) && (SHpnt->host_busy >= SHpnt->can_queue))
@@ -273,6 +275,8 @@ q = &SDpnt->request_queue; q->request_fn (q); } + +up (&SHpnt->host_queue_sema); }/*
@@ -285,6 +289,7 @@*/ all_clear =1 ;if (SHpnt->some_device_starved) { +down (&SHpnt->host_queue_sema);for (SDpnt = SHpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {request_queue_t *q; if ((SHpnt->can_queue > 0 && (SHpnt->host_busy >= SHpnt->can_queue))
@@ -299,6 +304,7 @@ q->request_fn (q); all_clear =0 ; } +up (&SHpnt->host_queue_sema);if (SDpnt ==NULL && all_clear) { SHpnt->some_device_starved =0 ; }
@@ -1117,8 +1123,13 @@ SHpnt->host_self_blocked = FALSE;/* Now that we are unblocked, try to start the queues. */ + +down (&SHpnt->host_queue_sema); +for (SDloop = SHpnt->host_queue; SDloop; SDloop = SDloop->next)scsi_queue_next_request (&SDloop->request_queue,NULL ); + +up (&SHpnt->host_queue_sema); }/*
@@ -1145,12 +1156,17 @@void scsi_report_bus_reset (struct Scsi_Host * SHpnt,int channel) {Scsi_Device *SDloop; + +down (&SHpnt->host_queue_sema); +for (SDloop = SHpnt->host_queue; SDloop; SDloop = SDloop->next) {if (channel == SDloop->channel) { SDloop->was_reset =1 ; SDloop->expecting_cc_ua =1 ; } } + +up (&SHpnt->host_queue_sema); }/*
drivers/scsi/scsi_scan.c +9 -1
@@ -559,6 +559,8 @@/* * Add it to the end of the shost->host_queue list. */ +down (&shost->host_queue_sema); +if (shost->host_queue !=NULL ) { sdev->prev = shost->host_queue;while (sdev->prev->next !=NULL )
@@ -567,6 +569,8 @@ }else shost->host_queue = sdev; +up (&shost->host_queue_sema); + }return (sdev); }
@@ -583,8 +587,12 @@ {if (sdev->prev !=NULL ) sdev->prev->next = sdev->next; -else +else { +down (&sdev->host->host_queue_sema); sdev->host->host_queue = sdev->next; +up (&sdev->host->host_queue_sema); + } +if (sdev->next !=NULL ) sdev->next->prev = sdev->prev;
include/linux/scsi_hotswap.h +84 -0
84 lines not on this page · raw message
Quoted 2 lines · Steven Dake
This patch has been reviewed by Alan Cox, Greg KH, Christoph Hellwig, Patrick Mansfield, Rob Landly, Jeff Garzik, Scott Murray, James
Glanced at briefly once, not reviewed.
Quoted 6 lines · Alan Cox
Quoted 2 lines · Steven Dake
This patch has been reviewed by Alan Cox, Greg KH, Christoph Hellwig, Patrick Mansfield, Rob Landly, Jeff Garzik, Scott Murray, James
Glanced at briefly once, not reviewed.
I'm going to leave the merging of this to the scsi people, in particular James and Doug. My personal feeling right now is that it's not going in the feature freeze, but as a driver thing I'm also convinced that especially if vendors need it, they'll add it anyway - and drivers tend to be less "frozen" than core code anyway (by necessity: we've always had to accept new drivers even in stable series).
Linus
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Alan Cox
Quoted 2 lines · Steven Dake
This patch has been reviewed by Alan Cox, Greg KH, Christoph Hellwig, Patrick Mansfield, Rob Landly, Jeff Garzik, Scott Murray, James
Quoted 8 lines
Quoted 1 line
Glanced at briefly once, not reviewed.
I'm going to leave the merging of this to the scsi people, in particular James and Doug. My personal feeling right now is that it's not going in the feature freeze, but as a driver thing I'm also convinced that especially if vendors need it, they'll add it anyway - and drivers tend to be less "frozen" than core code anyway (by necessity: we've always had to accept new drivers even in stable series).
My personal view is that it might be a candidate for inclusion in the future, but not in the current version. Several valid points have been raised by other people, so I'll not rehash those. Some of my own points include the fact that I detest the locking scheme this thing uses.
If you look at scsi_hotswap_insert_by_scsi_id() you see this:
down_interruptible (&scsi_host->host_queue_sema)
walk scsi_host->host_queue looking for a device with same id
up (&scsi_host->host_queue_sema)
did we find a device? If so, return, else call scan_scsis()
The problem is that scan_scsis() is where we would actually add the device to the scsi_host->host_queue and it's not inside the lock, so putting the check inside of a lock is a total waste of time. To prevent against races on insertion, you would need to, in the case that no device was present, add a device to the host_queue *while still holding the lock* in order to keep a second invocation of insert_by_scsi_id() from attempting to add the same device twice. Specifically, I'm thinking that a fiber channel controller that has a device flutter on and off the fiber bus can easily hit this problem. Imagine if you will:
fiber loop notices new device come up
driver calls insert_by_scsi_id to add drive
insert calls scan_scsis() to scan device
before scan completes drive flutters back off the fiber
driver calls remove_by_scsi_id
remove code doesn't find a valid device because original scan_scsis hasn't added it yet
drive comes back on fiber
driver calls insert_by_scsi_id
the check for drive present shows us clear because original scan_scsis hasn't added drive yet, so we call scan_scsis() again
we are now trying to scan drive in two different threads of execution, no locking against double addition of the same device, boom.
Nope, it's not there yet.
Signature · 4 lines
Doug Ledford <[email protected]> 919-754-3700 x44233 Red Hat, Inc. 1801 Varsity Dr. Raleigh, NC 27606
Quoted 5 lines · Alan Cox
Quoted 2 lines · Steven Dake
This patch has been reviewed by Alan Cox, Greg KH, Christoph Hellwig, Patrick Mansfield, Rob Landly, Jeff Garzik, Scott Murray, James
Glanced at briefly once, not reviewed.
I didn't look at it all, in fact. I just commented on some general hotplug issues raised in the discussion.
Scott
Signature · 4 lines
Scott Murray SOMA Networks, Inc. Toronto, Ontario e-mail: [email protected]
Umm, stop.
Scsi midlayer patches don't go directly to Linus, but through the linux-scsi list and James into the linux-scsi bk repository first.
The patch still has a bunch of problem not solved, and contains two things that should be independant patches.
The first patch should be the host_queue locking you added, this one currently has the following issues:
* you call spin_lock on a semaphore once!
* you take semaphores inside spinlocks and with interrupts disabled
* the coding style needs some imnprovements (you adds lots of empty lines, and there's a space before the opening brakes of function calls).
the actual driver still has other issues:
* you still duplicated lots of code from scsi.c
* your header is still in include/linux instead of include/scsi, but imho it should be merged into scsi.h anyway
* you still havent explain why wwn -> host id translation can't be done in userspace
* you still have useage information in the driverfs files.
Christoph,
I'll seperate the patches and fix the things you mention. I'm still not entirely pleased with the locking and Patrick Mansfield has suggested a better method for traversing the host_queue structure that I may consider.
Regarding WWN translation in userspace, there are several reasons why this apporach is inferior:
1. When a device is "inserted" into the system and isn't present, there is no way to map WWNs to target ids. Processing in userspace would work for removals, but what about insertions? Sure the driver could export its list of known WWNs (the firmware maintains a list of active WWNs on the bus and their associated scsi target, even if the SCSI subsystem doesn't know the device is there), but read #2 below:
2. There is no simple way to know what WWN maps to which SCSI ID from user space.
Typical FibreChannel drivers maintain this information, but it isn't exported to user space in any useable way. I suppose a list of device->WWN maps could be present in /proc or driverfs but that adds alot of string processing from user space and requires a standard format between drivers to be generically usable. Further, what if two drivers want to keep track of their device to WWN map? They must each register with some function that exports the list to userspace. Why go through all the trouble when the driver can just do the work?
3. Hotswap operations should be fast (10 msec or so) and userspace processing is slow. Compare two operations one using the defined hotswap mapping interface and one processing a file in proc processing in userspace:
while (not_eol) {
read syscall
convert string WWNS to 64 bit number
compare WWNS
if matched, convert string scsi target information (host, channel, target, lun) into 32 bit numbers
issue hotswap command using host channel target lun information
}
processing in kernel:
call wwn to scsi target converter in driver
in driver, each WWN is compared against wwn in WWN list if match found, scsi target information returned
Now, assume the above with 32 or more devices. That is alot of syscalls using userspace methodology (and slow). Let the kernel do the conversion directly. It has direct access to the data. Another choice would be to put a 64 bit field for wwn's into the scsi device structure. Then the scsi device list could be traveresed to find the correct target to hotswap. Would this be preferrrable to the interface currently implemented?
4. Having to map WWNs to scsi IDs in userspace is harder to use
its alot easier to remove a device to echo "wwn" > /sys/bus/scsi/hotswap_comamnds/remove_by_fc_wwn_wildcard
compare that with writing an application to process /proc entries (or something similiar).
other comments below:
Quoted 5 lines · Christoph Hellwig
Umm, stop.
Scsi midlayer patches don't go directly to Linus, but through the linux-scsi list and James into the linux-scsi bk repository first.
Quoted 9 lines
The patch still has a bunch of problem not solved, and contains two things that should be independant patches.
The first patch should be the
host_queuelocking you added, this one currently has the following issues:* you call
spin_lockon a semaphore once!
sloppy I'll fix.
Quoted 2 lines
* you take semaphores inside spinlocks and with interrupts disabled
good point I think I need to look at this patch more.
Quoted 7 lines
* the coding style needs some imnprovements (you adds lots of empty lines, and there's a space before the opening brakes of function calls).
the actual driver still has other issues:
* you still duplicated lots of code from
scsi.c
I'll remove the code from scsi.c and have it call the hotswap interfaces directly.
Quoted 3 lines
* your header is still in
include/linuxinstead ofinclude/scsi, but imho it should be merged intoscsi.hanyway
I'll merge into scsi.h.
Quoted 3 lines
* you still havent explain why wwn -> host id translation can't be done in userspace
see above
Quoted 4 lines
* you still have useage information in the driverfs files.
i'll remove.
Thanks for the great comments. -steve
Quoted 3 lines · [email protected] said:
This patch has been reviewed by Alan Cox, Greg KH, Christoph Hellwig, Patrick Mansfield, Rob Landly, Jeff Garzik, Scott Murray, James
Bottomley, Mike Anderson, Randy Dunlap and Patrick Mochel.
Well, I reviewed it but my though was that it should be built on the emerging hotplug infrastructure.
I'm currently trying to move things like this into the user layer, so from a design principle I don't want to have 90% using the user space stuff and 10% using its in-kernel equivalent because that means we have two mechanisms to maintain, thus doubling the work.
The problem you had with this was the 10ms requirement from removal notification to removal completion. Several people have already suggested that if that really is a hard and fast requirement, then you could simply treat the removal as a surprise removal rather than a planned one and work on fixing our surprise removal problems instead.
The bottom line is that I'm not convinced this can't be done using the existing infrastructure or a generic enhancement to it.
James
Quoted 2 lines · Steven Dake
This patch has been reviewed by Alan Cox, Greg KH, Christoph Hellwig, Patrick Mansfield, Rob Landly, Jeff Garzik, Scott Murray, James
I haven't got a single system around here with SCSI hardware in it anymore, and the last time I dealt with fibrechannel was almost two years ago now. I just tracked the patch for other people who would be interested.
Rob
P.S. Personally, I think SCSI is kind of pointless these days. I fit 20 IDE drives in a box once, configured as raid 5. These days that would be over 3 terabytes. Yes, the cables were annoying. And if you need more space than that just use a clustering filesystem over gigabit eithernet. Or for that matter, 100baseT with gigabit uplinks on the switches is usually plenty. (The cost of gigabit has come down, but the switches are murder. That's why bonding pairs of 100baseT cards isn't necessarily cost effective: the cards themselves are dirt cheap but switches with gigabit uplinks more than make up the difference, as we wait for 10gigE to bring the cost of normal 1gigE down to something reasonable...)
Then again I've always been trying to optimize the price to performance ratio more than anything else. I'm a cheap bastard and proud of it...
Signature · 2 lines
http://penguicon.sf.net - Terry Pratchett, Eric Raymond, Pete Abrams, Illiad, CmdrTaco, liquid nitrogen ice cream, and caffienated jello. Well why not?