File Upload to S3 Using Rest Api Imorti
Introduction
Recently I have meet a new requirement where we need to supercede an Oracle DB with AWS setup. And then nosotros will drop data in CSV format into AWS S3 and from at that place nosotros utilize AWS GLUE crawlers and ETL job to transform information to parquet format and share information technology with Amazon Redshift Spectrum to query the information using standard SQL or Apache Hive . At that place are multiple AWS connectors available in market for uploading data to AWS S3 from middleware similar SAP PO or SAP CPI. But here I want to share a simple UDF which can help yous with uploading any data to AWS S3 from SAP PO using Remainder API of AWS.
Main
AWS REST API usage require following header parameters which needs to be passed every time whenever y'all are calling to the endpoint. For more details regarding the headers, you can read AWS documentation.
- Authorization
- ten-amz-content-sha256
- content-length
- x- amz -date
- Host
- x- amz -storage-class
All these headers can but be generated by some scripts as they are dependent on runtime variables like payload, date and time etc. you tin go through the java classes which can generate these headers.
AWS S3 Remainder API has certain format for endpoint every bit well. So we volition generate endpoint using the same UDF.
Nosotros have below input parameters for the UDF.
- bucketName : AWS S3 Bucket name as provided past the admin
- regionName : AWS S3 saucepan region ( eg . usa-e-1 )
- awsAccessKey : AWS IAM user Admission key
- awsSecretKey : AWS IAM user Scecret Key
- objectContent : Payload which should exist passed as Residuum Trunk
- path : Folder details with filename proper noun (/ Folder_Name /Filename.txt)
Deasign object Evolution:
Now nosotros can kickoff ESR design object evolution. Firstly, nosotros need to create a Function library which tin can exist used as UDF in the mapping.
Function Library
Create a function Signature with following input arguments and add below entries under Import Instructions.
com.sap.aii.mapping.api.* , com.sap.aii.mapping .lookup.* , com.sap.aii.mappingtool.tf7.rt.* , java.io.* , coffee.lang .reflect.* , java.util.* , java.cyberspace.* , javax.crypto .spec.SecretKeySpec , javax.crypto .Mac , coffee.security .MessageDigest , coffee. text.SimpleDateFormat , javax.crypto .* , java.nio.charset .Charset
Then Add together beneath coffee code to function Signature.
URL endpointUrl; attempt { if (regionName.equals("u.s.a.-east-one")) { endpointUrl = new URL("https://s3.amazonaws.com/" + bucketName + path); } else { endpointUrl = new URL("https://s3-" + regionName + ".amazonaws.com/" + bucketName + path); } } catch (MalformedURLException east) { throw new RuntimeException("Unable to parse service endpoint: " + e.getMessage()); } Cord Content = objectContent; // precompute hash of the body content byte[] contentHash = AWS4SignerBase.hash(Content); String contentHashString = BinaryUtils.toHex(contentHash); Map<String, String> headers = new HashMap<String, String>(); headers.put("x-amz-content-sha256", contentHashString); headers.put("content-length", "" + objectContent.getBytes(Charset.forName("UTF-viii")).length); headers.put("x-amz-storage-class", "REDUCED_REDUNDANCY"); AWS4SignerForAuthorizationHeader signer = new AWS4SignerForAuthorizationHeader( endpointUrl, "PUT", "s3", regionName); Cord authorization = signer.computeSignature(headers, null, // no query parameters contentHashString, awsAccessKey, awsSecretKey); // express say-so for this as a header headers.put("Authorization", authorization); String var1 = headers.get("ten-amz-content-sha256"); String var2 = headers.get("x-amz-date"); String var3 = endpointUrl.toString(); Cord var4 = headers.get("content-length"); DynamicConfiguration conf1 = (DynamicConfiguration) container.getTransformationParameters().get(StreamTransformationConstants.DYNAMIC_CONFIGURATION); //DynamicConfigurationKey key1 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/Xi/System/File","FileName"); DynamicConfigurationKey key2 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/Xi/System/Remainder","Auth"); conf1.put(key2,authorization); DynamicConfigurationKey key3 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","xamzcontent"); conf1.put(key3,var1); DynamicConfigurationKey key4 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","xamzdate"); conf1.put(key4,var2); DynamicConfigurationKey key5 = DynamicConfigurationKey.create( "http:/"+"/sap.com/eleven/11/System/REST","endpoint"); conf1.put(key5,var3); DynamicConfigurationKey key6 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/Xi/System/Remainder","len"); conf1.put(key6,var4); render(Content); And then add beneath required nested classes in the Attributes and Methods surface area.
public abstract static form AWS4SignerBase { /** SHA256 hash of an empty request body **/ public static final Cord EMPTY_BODY_SHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; public static last Cord UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD"; public static concluding String SCHEME = "AWS4"; public static last Cord ALGORITHM = "HMAC-SHA256"; public static final String TERMINATOR = "aws4_request"; /** format strings for the date/fourth dimension and appointment stamps required during signing **/ public static terminal String ISO8601BasicFormat = "yyyyMMdd'T'HHmmss'Z'"; public static terminal String DateStringFormat = "yyyyMMdd"; protected URL endpointUrl; protected Cord httpMethod; protected String serviceName; protected String regionName; protected concluding SimpleDateFormat dateTimeFormat; protected concluding SimpleDateFormat dateStampFormat; /** * Create a new AWS V4 signer. * * @param endpointUri * The service endpoint, including the path to whatever resource. * @param httpMethod * The HTTP verb for the request, e.g. GET. * @param serviceName * The signing name of the service, due east.g. 's3'. * @param regionName * The organization proper noun of the AWS region associated with the * endpoint, e.g. usa-east-i. */ public AWS4SignerBase(URL endpointUrl, String httpMethod, String serviceName, String regionName) { this.endpointUrl = endpointUrl; this.httpMethod = httpMethod; this.serviceName = serviceName; this.regionName = regionName; dateTimeFormat = new SimpleDateFormat(ISO8601BasicFormat); dateTimeFormat.setTimeZone(new SimpleTimeZone(0, "UTC")); dateStampFormat = new SimpleDateFormat(DateStringFormat); dateStampFormat.setTimeZone(new SimpleTimeZone(0, "UTC")); } /** * Returns the canonical collection of header names that volition be included in * the signature. For AWS4, all header names must be included in the process * in sorted canonicalized order. */ protected static String getCanonicalizeHeaderNames(Map<String, String> headers) { List<String> sortedHeaders = new ArrayList<String>(); sortedHeaders.addAll(headers.keySet()); Collections.sort(sortedHeaders, String.CASE_INSENSITIVE_ORDER); StringBuilder buffer = new StringBuilder(); for (String header : sortedHeaders) { if (buffer.length() > 0) buffer.append(";"); buffer.append(header.toLowerCase()); } return buffer.toString(); } /** * Computes the canonical headers with values for the request. For AWS4, all * headers must be included in the signing process. */ protected static String getCanonicalizedHeaderString(Map<String, String> headers) { if ( headers == nil || headers.isEmpty() ) { return ""; } // step1: sort the headers past case-insensitive lodge List<String> sortedHeaders = new ArrayList<String>(); sortedHeaders.addAll(headers.keySet()); Collections.sort(sortedHeaders, String.CASE_INSENSITIVE_ORDER); // step2: form the canonical header:value entries in sorted club. // Multiple white spaces in the values should exist compressed to a single // space. StringBuilder buffer = new StringBuilder(); for (String key : sortedHeaders) { buffer.append(key.toLowerCase().replaceAll("\\s+", " ") + ":" + headers.go(key).replaceAll("\\s+", " ")); buffer.suspend("\due north"); } return buffer.toString(); } /** * Returns the approved asking string to get into the signer process; this consists of several approved sub-parts. * @return */ protected static Cord getCanonicalRequest(URL endpoint, String httpMethod, String queryParameters, String canonicalizedHeaderNames, String canonicalizedHeaders, String bodyHash) { String canonicalRequest = httpMethod + "\n" + getCanonicalizedResourcePath(endpoint) + "\north" + queryParameters + "\n" + canonicalizedHeaders + "\due north" + canonicalizedHeaderNames + "\n" + bodyHash; return canonicalRequest; } /** * Returns the canonicalized resource path for the service endpoint. */ protected static String getCanonicalizedResourcePath(URL endpoint) { if ( endpoint == null ) { return "/"; } String path = endpoint.getPath(); if ( path == null || path.isEmpty() ) { return "/"; } Cord encodedPath = HttpUtils.urlEncode(path, true); if (encodedPath.startsWith("/")) { return encodedPath; } else { render "/".concat(encodedPath); } } /** * Examines the specified query string parameters and returns a * canonicalized class. * <p> * The canonicalized query string is formed by first sorting all the query * string parameters, then URI encoding both the cardinal and value and then * joining them, in order, separating key value pairs with an '&'. * * @param parameters * The query cord parameters to be canonicalized. * * @render A canonicalized form for the specified query string parameters. */ public static String getCanonicalizedQueryString(Map<String, String> parameters) { if ( parameters == null || parameters.isEmpty() ) { render ""; } SortedMap<Cord, String> sorted = new TreeMap<String, String>(); Iterator<Map.Entry<Cord, Cord>> pairs = parameters.entrySet().iterator(); while (pairs.hasNext()) { Map.Entry<String, String> pair = pairs.next(); String key = pair.getKey(); String value = pair.getValue(); sorted.put(HttpUtils.urlEncode(key, imitation), HttpUtils.urlEncode(value, false)); } StringBuilder architect = new StringBuilder(); pairs = sorted.entrySet().iterator(); while (pairs.hasNext()) { Map.Entry<String, String> pair = pairs.next(); builder.suspend(pair.getKey()); builder.suspend("="); architect.append(pair.getValue()); if (pairs.hasNext()) { builder.suspend("&"); } } return builder.toString(); } protected static Cord getStringToSign(String scheme, String algorithm, String dateTime, String telescopic, Cord canonicalRequest) { Cord stringToSign = scheme + "-" + algorithm + "\n" + dateTime + "\due north" + scope + "\n" + BinaryUtils.toHex(hash(canonicalRequest)); return stringToSign; } /** * Hashes the string contents (causeless to exist UTF-8) using the SHA-256 * algorithm. */ public static byte[] hash(Cord text) { endeavour { MessageDigest doc = MessageDigest.getInstance("SHA-256"); md.update(text.getBytes("UTF-8")); return md.digest(); } take hold of (Exception e) { throw new RuntimeException("Unable to compute hash while signing request: " + e.getMessage(), e); } } /** * Hashes the byte assortment using the SHA-256 algorithm. */ public static byte[] hash(byte[] data) { try { MessageDigest physician = MessageDigest.getInstance("SHA-256"); doctor.update(information); return md.digest(); } catch (Exception e) { throw new RuntimeException("Unable to compute hash while signing request: " + e.getMessage(), e); } } protected static byte[] sign(String stringData, byte[] key, String algorithm) { try { byte[] information = stringData.getBytes("UTF-8"); Mac mac = Mac.getInstance(algorithm); mac.init(new SecretKeySpec(primal, algorithm)); return mac.doFinal(information); } catch (Exception east) { throw new RuntimeException("Unable to calculate a request signature: " + e.getMessage(), e); } } } public static form AWS4SignerForAuthorizationHeader extends AWS4SignerBase { public AWS4SignerForAuthorizationHeader(URL endpointUrl, Cord httpMethod, String serviceName, Cord regionName) { super(endpointUrl, httpMethod, serviceName, regionName); } /** * Computes an AWS4 signature for a request, ready for inclusion as an * 'Authorization' header. * * @param headers * The request headers; 'Host' and '10-Amz-Appointment' volition exist added to * this fix. * @param queryParameters * Whatsoever query parameters that will exist added to the endpoint. The * parameters should be specified in canonical format. * @param bodyHash * Precomputed SHA256 hash of the request body content; this * value should also be ready every bit the header 'X-Amz-Content-SHA256' * for not-streaming uploads. * @param awsAccessKey * The user's AWS Admission Key. * @param awsSecretKey * The user's AWS Hush-hush Key. * @return The computed authority string for the asking. This value * needs to exist ready as the header 'Authority' on the subsequent * HTTP request. */ public String computeSignature(Map<String, String> headers, Map<String, String> queryParameters, String bodyHash, String awsAccessKey, String awsSecretKey) { // beginning get the date and time for the subsequent request, and convert // to ISO 8601 format for use in signature generation Date now = new Date(); String dateTimeStamp = dateTimeFormat.format(now); // update the headers with required 'x-amz-engagement' and 'host' values headers.put("10-amz-engagement", dateTimeStamp); String hostHeader = endpointUrl.getHost(); int port = endpointUrl.getPort(); if ( port > -1 ) { hostHeader.concat(":" + Integer.toString(port)); } headers.put("Host", hostHeader); // canonicalize the headers; we demand the set of header names as well equally the // names and values to become into the signature process String canonicalizedHeaderNames = getCanonicalizeHeaderNames(headers); Cord canonicalizedHeaders = getCanonicalizedHeaderString(headers); // if any query cord parameters have been supplied, canonicalize them String canonicalizedQueryParameters = getCanonicalizedQueryString(queryParameters); // canonicalize the various components of the asking String canonicalRequest = getCanonicalRequest(endpointUrl, httpMethod, canonicalizedQueryParameters, canonicalizedHeaderNames, canonicalizedHeaders, bodyHash); Arrangement.out.println("--------- Canonical request --------"); System.out.println(canonicalRequest); System.out.println("------------------------------------"); // construct the string to be signed Cord dateStamp = dateStampFormat.format(now); String scope = dateStamp + "/" + regionName + "/" + serviceName + "/" + TERMINATOR; String stringToSign = getStringToSign(SCHEME, ALGORITHM, dateTimeStamp, scope, canonicalRequest); Organization.out.println("--------- String to sign -----------"); System.out.println(stringToSign); Arrangement.out.println("------------------------------------"); // compute the signing cardinal byte[] kSecret = (SCHEME + awsSecretKey).getBytes(); byte[] kDate = sign(dateStamp, kSecret, "HmacSHA256"); byte[] kRegion = sign(regionName, kDate, "HmacSHA256"); byte[] kService = sign(serviceName, kRegion, "HmacSHA256"); byte[] kSigning = sign(TERMINATOR, kService, "HmacSHA256"); byte[] signature = sign(stringToSign, kSigning, "HmacSHA256"); String credentialsAuthorizationHeader = "Credential=" + awsAccessKey + "/" + scope; String signedHeadersAuthorizationHeader = "SignedHeaders=" + canonicalizedHeaderNames; String signatureAuthorizationHeader = "Signature=" + BinaryUtils.toHex(signature); String authorizationHeader = SCHEME + "-" + ALGORITHM + " " + credentialsAuthorizationHeader + ", " + signedHeadersAuthorizationHeader + ", " + signatureAuthorizationHeader; return authorizationHeader; } } public static form BinaryUtils { /** * Converts byte data to a Hex-encoded cord. * * @param information * data to hex encode. * * @return hex-encoded string. */ public static String toHex(byte[] data) { StringBuilder sb = new StringBuilder(information.length * 2); for (int i = 0; i < data.length; i++) { String hex = Integer.toHexString(data[i]); if (hex.length() == 1) { // Append leading zero. sb.append("0"); } else if (hex.length() == 8) { // Remove ff prefix from negative numbers. hex = hex.substring(6); } sb.suspend(hex); } render sb.toString().toLowerCase(Locale.getDefault()); } /** * Converts a Hex-encoded data cord to the original byte information. * * @param hexData * hex-encoded information to decode. * @return decoded data from the hex cord. */ public static byte[] fromHex(String hexData) { byte[] consequence = new byte[(hexData.length() + 1) / 2]; String hexNumber = aught; int stringOffset = 0; int byteOffset = 0; while (stringOffset < hexData.length()) { hexNumber = hexData.substring(stringOffset, stringOffset + 2); stringOffset += 2; result[byteOffset++] = (byte) Integer.parseInt(hexNumber, 16); } return result; } } public static course HttpUtils { /** * Makes a http request to the specified endpoint */ public static String invokeHttpRequest(URL endpointUrl, String httpMethod, Map<String, String> headers, Cord requestBody) { HttpURLConnection connexion = createHttpConnection(endpointUrl, httpMethod, headers); try { if ( requestBody != null ) { DataOutputStream wr = new DataOutputStream( connection.getOutputStream()); wr.writeBytes(requestBody); wr.affluent(); wr.shut(); } } grab (Exception e) { throw new RuntimeException("Request failed. " + e.getMessage(), e); } return executeHttpRequest(connexion); } public static String executeHttpRequest(HttpURLConnection connection) { try { // Get Response InputStream is; try { is = connection.getInputStream(); } take hold of (IOException e) { is = connexion.getErrorStream(); } BufferedReader rd = new BufferedReader(new InputStreamReader(is)); String line; StringBuffer response = new StringBuffer(); while ((line = rd.readLine()) != zero) { response.suspend(line); response.suspend('\r'); } rd.close(); return response.toString(); } catch (Exception due east) { throw new RuntimeException("Request failed. " + e.getMessage(), east); } finally { if (connection != zero) { connection.disconnect(); } } } public static HttpURLConnection createHttpConnection(URL endpointUrl, String httpMethod, Map<String, String> headers) { try { HttpURLConnection connectedness = (HttpURLConnection) endpointUrl.openConnection(); connection.setRequestMethod(httpMethod); if ( headers != null ) { System.out.println("--------- Request headers ---------"); for ( String headerKey : headers.keySet() ) { System.out.println(headerKey + ": " + headers.become(headerKey)); connection.setRequestProperty(headerKey, headers.get(headerKey)); } } connection.setUseCaches(imitation); connection.setDoInput(true); connexion.setDoOutput(truthful); return connexion; } grab (Exception e) { throw new RuntimeException("Cannot create connection. " + e.getMessage(), e); } } public static String urlEncode(String url, boolean keepPathSlash) { String encoded; effort { encoded = URLEncoder.encode(url, "UTF-viii"); } catch (UnsupportedEncodingException e) { throw new RuntimeException("UTF-8 encoding is not supported.", e); } if ( keepPathSlash ) { encoded = encoded.replace("%2F", "/"); } return encoded; } } Then nosotros tin can import this FL to any of the Message mapping and laissez passer required input parameters to the UDF. This UDF will automatically ready required headers and endpoint URL in the ASMA of REST adapter which can exist accessed at runtime.
UDF
This UDF volition return whatever parameter you accept passed in the input parameter " objectContent ".
And then you can create a XSLT mapping to fetch only that field and pass it to Residual torso. (you can also use module configuration to reach the same.)
XSLT Mapping to convert XML to text
Configuration object Development
Now nosotros tin can develop Configuration Objects in ID. I am just explaining what are required changes in Remainder receiver channel.
Firstly, we take to admission the ASMA parameters from runtime and set them as REST headers. Delight follow the below screenshot and add as per that. Also add together PUT operation in REST Operation tab.
Rest Endpoint configuration
Rest Headers configuration
Residual request format configuration
Aqueduct Module configuration
You demand to fix CRLF to LF converter module if y'all are using XSLT mapping to set payload as Residuum body as some CR volition be added in the conversion process. If y'all apply xml to apartment file converter module, so you lot practise not accept to add that module. Other modules are simply used for logging purposes.
Test Results
You can find 200 response code in the audit logs.
Audit logs
AWS S3 Panel
Conclusion
All the java codes are provided by AWS documentation . I have implemented the same with some modification. It's a custom code so it requires much maintenance effort to fix if any issue occurs so I suggest to use AWS S3 connectors merely you can always explore and it's a bit less costly than the connectors. You can as well use the same java code and make a jar file and use it in SAP CPI and apply one groovy script to make a function.
Source: https://blogs.sap.com/2020/10/20/file-upload-in-aws-s3-using-rest-api/