import numpy as np caffe_root = "/home/your_name/Downloads/caffe-master/" import sys sys.path.insert(0, caffe_root+"python") import os import caffe import io
caffe.set_mode_cpu() MODEL_FILE = caffe_root + "models/colon_caffenet/colon_deploy.prototxt" #change the deploy file same as the train_val.prototxt PRETRAINED = caffe_root + "models/colon_caffenet/caffenet_train_iter_80.caffemodel" MEAN = caffe_root + "examples/colon/colon_mean.npy" # later I will tell how to tranform *.binaryproto to *.npy net = caffe.Classifier(MODEL_FILE, PRETRAINED, mean = np.load(MEAN).mean(1).mean(1), channel_swap=(2,1,0), raw_scale=255, image_dims=(256, 256))
filewriter = open(caffe_root+"data/colon/test_result.txt","w+") for root,dirs,files in os.walk(caffe_root+"data/colon/test/"): # all the images are in test folder for file in files IMAGE_FILE = os.path.join(root,file).decode('gbk').encode('utf8') input_image = caffe.io.load_image(IMAGE_FILE) prediction = net.predict([input_image]) string = os.path.basename(IMAGE_FILE)+" "+str(prediction.argmax())+"\n" filewriter(string) print os.path.basename(IMAGE_FILE), prediction.argmax()
convert binaryproto to npy
1 2 3 4 5 6 7 8 9 10 11 12 13 14
import caffe import numpy as np import sys
if len(sys.argv) != 3: print"Usage: python convert_protomean.py proto.mean out.npy" sys.exit()
blob = caffe.proto.caffe_pb2.BlobProto() data = open( sys.argv , 'rb' ).read() blob.ParseFromString(data) arr = np.array( caffe.io.blobproto_to_array(blob) ) out = arr np.save( sys.argv , out )
Meets lots of problems when import caffe
When python executes “import caffe”, many problems comes out. All problems are about the python packages. Although I have set up the python library pointing to anaconda2 lib, it seems programme will find all the dependencies in /usr/lib/python2.7/dist-packages. So if you have encountered the same issues, copy all dependencies in /anaconda2/lib/python2.7/site-packages to /usr/lib/python2.7/dist-packages
Since I have installed Caffe, here I will adopt it to my own application. The aim of my examples is for binary classification of images. I will exploit the trained bvlc_reference_caffenet model and then fine tune it for my application.
The whole steps are as follows:
Prepare original data
You need to prepare four files:
train folder which contains the training images
val folder which contains the testing images
train.txt file which contains the labels of training images.
val.txt file which contains the labels of testing images.
Note that the order of image name in *.txt file is equal to that in train folder and val folder.
The train.txt looks like:
That means train folder includes two subfolders cat and dog. In each subfolder, it contains the cat or dog images relatively.
What you need to do is going to caffe_master/data folder, creating a new folder named myself, then putting the four files above into it.
So now in myself folder, you can see four files: train, val, train.txt, val.txt
Transform data format
Later we will use some tools to transform the image files to the data format which ImageNet model consumes.
Copy all *.sh files in caffe-master/examples/imagenet to myself folder.
Change the file path in create_imagenet.sh
Run it and then it will generate myself_train_lmdb and myself_val_lmdb in myself
Compute the mean value
Change the file path in make_imagenet_mean.sh
Run it and it will generate myself_mean.binaryproto in myself
Okay, till now, you have prepared all the data the ImageNet Model needs.
Under myself folder, you can see:
myself_train_lmdb folder: it contains the training data
myself_val_lmdb folder: it contains the testing data
myself_mean.binaryproto: it is the mean value
Fine Tune the trained model
Firstly, we need to download the trained model.
# the root path is "caffe-master" ./scripts/download_model_binary.py models/bvlc_reference_caffenet # it will take some time to download it
After that you need to fine tune it.
change the train_val.prototxt (1) change the input data path relatively
(2) We change the name of last layer from “fc8” to “fc8_myself”. Because there is no layer named “fc8_myself”, it’s weights will be retrained according to random weights.
These days, I am in the middle of the configuration of Caffe in Ubuntu14.04 due to the need of a small project. Here I want to write down the steps to show how to install it.
If you follow the installation instructions of the official site， I think you will be crazy. The documents are not in order. I succeeded to install it according to a Chinese blog. Apart from simple translation, I will also add some contents.
Basically, I followed this blog to deploy the clusters’ environment. Instead of using spark 1.3.0, we adopt the latest version 1.5.1. But it doesn’t influence the configuration.
The issue maybe you would come across
I met a problem in the process of free-password login with SSH. Two of machines could access each other. Only one machine got trapped that other two machines couldn’t access it. After some investigation, we found it was due to the authority.
The authority of .ssh folder must be confined to 755
The authority of authorized_keys must be confined to 600